[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1005822372 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003393] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005022] kvm-guest: setup PV IPIs [ 0.010947] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.011000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.011027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012017] pid_max: default: 32768 minimum: 301 [ 0.013757] LSM: Security Framework initializing [ 0.015008] Yama: becoming mindful. [ 0.016094] SELinux: Initializing. [ 0.017105] *** VALIDATE selinux *** [ 0.034451] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.043000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.043171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.044165] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.046086] *** VALIDATE tmpfs *** [ 0.049096] *** VALIDATE proc *** [ 0.051041] *** VALIDATE cgroup *** [ 0.052014] *** VALIDATE cgroup2 *** [ 0.053383] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.055061] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.057006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.058040] Spectre V2 : User space: Vulnerable [ 0.060011] Speculative Store Bypass: Vulnerable [ 0.064044] debug: unmapping init [mem 0xffffffffae259000-0xffffffffae260fff] [ 0.068000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.069886] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.070032] ... version: 2 [ 0.071036] ... bit width: 48 [ 0.072031] ... generic registers: 4 [ 0.073032] ... value mask: 0000ffffffffffff [ 0.074035] ... max period: 00007fffffffffff [ 0.075039] ... fixed-purpose events: 3 [ 0.076030] ... event mask: 000000070000000f [ 0.077861] rcu: Hierarchical SRCU implementation. [ 0.082202] smp: Bringing up secondary CPUs ... [ 0.084124] x86: Booting SMP configuration: [ 0.085046] .... node #0, CPUs: #1 #2 #3 [ 0.093030] smp: Brought up 1 node, 4 CPUs [ 0.095035] smpboot: Max logical packages: 1 [ 0.096041] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139023] node 0 deferred pages initialised in 40ms [ 0.151016] devtmpfs: initialized [ 0.157000] x86/mm: Memory block size: 128MB [ 0.172754] gcov: version magic: 0x41383552 [ 0.182527] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.183173] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.192689] pinctrl core: initialized pinctrl subsystem [ 0.201010] [ 0.203016] ************************************************************* [ 0.211027] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.219035] ** ** [ 0.228029] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.234025] ** ** [ 0.242025] ** This means that this kernel is built to expose internal ** [ 0.250024] ** IOMMU data structures, which may compromise security on ** [ 0.258025] ** your system. ** [ 0.266024] ** ** [ 0.272024] ** If you see this message and you are not debugging the ** [ 0.280023] ** kernel, report this immediately to your vendor! ** [ 0.288010] ** ** [ 0.294026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.303025] ************************************************************* [ 0.312979] NET: Registered protocol family 16 [ 0.316139] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.325243] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.336127] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.347422] cpuidle: using governor menu [ 0.356853] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.364313] PCI: Using configuration type 1 for base access [ 0.370347] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.398114] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.403035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.411040] cryptd: max_cpu_qlen set to 1000 [ 0.421291] ACPI: Added _OSI(Module Device) [ 0.426031] ACPI: Added _OSI(Processor Device) [ 0.432030] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.433025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.452895] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.475321] ACPI: Interpreter enabled [ 0.481188] ACPI: PM: (supports S0 S3 S4 S5) [ 0.482036] ACPI: Using IOAPIC for interrupt routing [ 0.483808] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.495000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.533576] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.540088] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.549072] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.559143] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.576075] acpiphp: Slot [2] registered [ 0.582217] acpiphp: Slot [5] registered [ 0.588248] acpiphp: Slot [6] registered [ 0.591506] acpiphp: Slot [7] registered [ 0.597428] acpiphp: Slot [8] registered [ 0.602410] acpiphp: Slot [9] registered [ 0.609397] acpiphp: Slot [10] registered [ 0.616000] acpiphp: Slot [3] registered [ 0.621300] acpiphp: Slot [4] registered [ 0.626246] acpiphp: Slot [11] registered [ 0.632148] acpiphp: Slot [12] registered [ 0.637235] acpiphp: Slot [13] registered [ 0.643000] acpiphp: Slot [14] registered [ 0.646459] acpiphp: Slot [15] registered [ 0.650212] acpiphp: Slot [16] registered [ 0.656771] acpiphp: Slot [17] registered [ 0.661237] acpiphp: Slot [18] registered [ 0.667900] acpiphp: Slot [19] registered [ 0.671234] acpiphp: Slot [20] registered [ 0.672179] acpiphp: Slot [21] registered [ 0.675171] acpiphp: Slot [22] registered [ 0.680000] acpiphp: Slot [23] registered [ 0.684170] acpiphp: Slot [24] registered [ 0.690464] acpiphp: Slot [25] registered [ 0.692169] acpiphp: Slot [26] registered [ 0.695183] acpiphp: Slot [27] registered [ 0.699385] acpiphp: Slot [28] registered [ 0.704155] acpiphp: Slot [29] registered [ 0.706236] acpiphp: Slot [30] registered [ 0.710241] acpiphp: Slot [31] registered [ 0.716100] PCI host bridge to bus 0000:00 [ 0.720035] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.724035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.730042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.735046] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.743059] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.751046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.756922] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.763060] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.774133] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.796000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.808083] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.817037] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.823032] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.835038] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.844020] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.849741] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.857084] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.863326] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 12695 usecs [ 0.877382] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.889000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.918000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.934025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.959378] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.981060] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.000040] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.039017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 1.069378] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.083000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.102035] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.136015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 1.168856] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.180000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.201040] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.235020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.260697] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.274000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.300024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.337014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.363337] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.375000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.392020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.434023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.471701] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.487000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.505025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.543020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.574929] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.582827] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.591570] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.599153] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.605488] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.616254] iommu: Default domain type: Passthrough [ 1.627402] SCSI subsystem initialized [ 1.630992] ACPI: bus type USB registered [ 1.634271] usbcore: registered new interface driver usbfs [ 1.639648] usbcore: registered new interface driver hub [ 1.644820] usbcore: registered new device driver usb [ 1.650237] pps_core: LinuxPPS API ver. 1 registered [ 1.656015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.665106] PTP clock support registered [ 1.671055] EDAC MC: Ver: 3.0.0 [ 1.675255] PCI: Using ACPI for IRQ routing [ 1.677837] NetLabel: Initializing [ 1.681020] NetLabel: domain hash size = 128 [ 1.685016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.690214] NetLabel: unlabeled traffic allowed by default [ 1.696302] vgaarb: loaded [ 1.700412] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.707024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.720564] clocksource: Switched to clocksource kvm-clock [ 1.978542] VFS: Disk quotas dquot_6.6.0 [ 1.987648] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.998182] *** VALIDATE ramfs *** [ 2.002128] *** VALIDATE hugetlbfs *** [ 2.008738] pnp: PnP ACPI init [ 2.016922] pnp: PnP ACPI: found 6 devices [ 2.058328] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 2.068397] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 2.076127] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 2.084750] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 2.092614] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 2.099907] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 2.109404] NET: Registered protocol family 2 [ 2.116236] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 2.130367] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 2.141631] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 2.154642] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 2.163808] TCP: Hash tables configured (established 65536 bind 65536) [ 2.171091] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 2.181528] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.191633] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.205461] NET: Registered protocol family 1 [ 2.214241] RPC: Registered named UNIX socket transport module. [ 2.223045] RPC: Registered udp transport module. [ 2.231969] RPC: Registered tcp transport module. [ 2.239018] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.246635] NET: Registered protocol family 44 [ 2.252119] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 2.257490] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 2.263575] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.272659] PCI: CLS 0 bytes, default 64 [ 2.281251] Unpacking initramfs... [ 5.973109] debug: unmapping init [mem 0xffff8a03fcc54000-0xffff8a03fffbffff] [ 6.017857] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 6.033908] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 6.045791] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.255405] Initialise system trusted keyrings [ 7.261314] Key type blacklist registered [ 7.267145] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.288524] zbud: loaded [ 7.295858] *** VALIDATE nfs *** [ 7.299483] *** VALIDATE nfs4 *** [ 7.304857] pstore: using deflate compression [ 7.318628] Platform Keyring initialized [ 7.558515] NET: Registered protocol family 38 [ 7.565146] Key type asymmetric registered [ 7.571705] Asymmetric key parser 'x509' registered [ 7.576336] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.586162] io scheduler mq-deadline registered [ 7.591823] io scheduler kyber registered [ 7.596309] io scheduler bfq registered [ 7.600910] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.609372] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.615109] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.622237] ACPI: Power Button [PWRF] [ 7.635525] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.653658] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.684111] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.704759] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.751357] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.795126] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.839531] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.850377] Non-volatile memory driver v1.3 [ 7.855241] Linux agpgart interface v0.103 [ 7.923586] virtio_blk virtio1: [vda] 134008 512-byte logical blocks (68.6 MB/65.4 MiB) [ 7.934787] vda: detected capacity change from 0 to 68612096 [ 7.983781] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.997617] vdb: detected capacity change from 0 to 1073741824 [ 8.036314] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.047030] vdc: detected capacity change from 0 to 2621440000 [ 8.089884] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.099526] vdd: detected capacity change from 0 to 2621440000 [ 8.131483] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.139692] vde: detected capacity change from 0 to 4294967296 [ 8.180489] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.187640] vdf: detected capacity change from 0 to 4294967296 [ 8.206349] libphy: Fixed MDIO Bus: probed [ 8.226013] usbcore: registered new interface driver usbserial_generic [ 8.237133] usbserial: USB Serial support registered for generic [ 8.243661] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.257181] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.263743] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.272057] mousedev: PS/2 mouse device common for all mice [ 8.283157] rtc_cmos 00:05: RTC can wake from S4 [ 8.294888] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.298143] rtc_cmos 00:05: registered as rtc0 [ 8.315928] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.324161] intel_pstate: CPU model not supported [ 8.329712] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.339223] hid: raw HID events driver (C) Jiri Kosina [ 8.345530] usbcore: registered new interface driver usbhid [ 8.350139] usbhid: USB HID core driver [ 8.350391] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.352705] drop_monitor: Initializing network drop monitor service [ 8.369055] Initializing XFRM netlink socket [ 8.375835] NET: Registered protocol family 10 [ 8.386462] Segment Routing with IPv6 [ 8.390799] NET: Registered protocol family 17 [ 8.396624] mpls_gso: MPLS GSO support [ 8.412353] RAS: Correctable Errors collector initialized. [ 8.420240] AVX version of gcm_enc/dec engaged. [ 8.426109] AES CTR mode by8 optimization enabled [ 8.623837] registered taskstats version 1 [ 8.629665] Loading compiled-in X.509 certificates [ 8.635734] zswap: loaded using pool lzo/zbud [ 8.693630] Key type big_key registered [ 8.721202] Key type encrypted registered [ 8.726933] ima: No TPM chip found, activating TPM-bypass! [ 8.731482] ima: Allocated hash algorithm: sha1 [ 8.736367] ima: No architecture policies found [ 8.741324] evm: Initialising EVM extended attributes: [ 8.746471] evm: security.selinux [ 8.749763] evm: security.ima [ 8.752651] evm: security.capability [ 8.756288] evm: HMAC attrs: 0x1 [ 8.761212] rtc_cmos 00:05: setting system clock to 2026-04-24 01:31:58 UTC (1776994318) [ 8.769928] Unstable clock detected, switching default tracing clock to "global" [ 8.769928] If you want to keep using the local clock, then add: [ 8.769928] "trace_clock=local" [ 8.769928] on the kernel command line [ 8.800115] debug: unmapping init [mem 0xffffffffaf203000-0xffffffffaf3fffff] [ 8.814097] debug: unmapping init [mem 0xffffffffadf82000-0xffffffffae258fff] [ 8.828121] Write protecting the kernel read-only data: 28672k [ 8.840420] debug: unmapping init [mem 0xffffffffac603000-0xffffffffac7fffff] [ 8.848224] debug: unmapping init [mem 0xffffffffacf14000-0xffffffffacffffff] [ 8.945755] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.979521] systemd[1]: Detected virtualization kvm. [ 8.984925] systemd[1]: Detected architecture x86-64. [ 8.993335] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.041658] systemd[1]: No hostname configured. [ 9.046180] systemd[1]: Set hostname to . [ 9.052969] random: systemd: uninitialized urandom read (16 bytes read) [ 9.060406] systemd[1]: Initializing machine ID from random generator. [ 9.192189] random: ln: uninitialized urandom read (6 bytes read) [ 9.454053] random: systemd: uninitialized urandom read (16 bytes read) [ 9.463852] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 9.480357] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 9.515206] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.062021] device-mapper: uevent: version 1.0.3 [ 11.068276] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ 11.691886] random: fast init done Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.649091] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 12.912079] scsi host0: ata_piix [ 12.960657] scsi host1: ata_piix [ 12.964089] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.972600] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.269616] random: crng init done [ 18.274070] random: 7 urandom warning(s) missed due to ratelimiting [ 18.933056] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.084508] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.878534] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.516528] SELinux: Disabled at runtime. [ 23.618185] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.647169] systemd[1]: Detected virtualization kvm. [ 23.652135] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.798956] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.807751] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.823705] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.838911] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.849891] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.876120] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.891502] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 25.419173] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.202826] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.895153] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.955411] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.225413] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.292783] EDAC sbridge: Ver: 1.1.2 [ 30.333977] Key type dns_resolver registered [ 30.637262] hrtimer: interrupt took 9740929 ns [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 32.679120] NFS: Registering the id_resolver key type [ 32.699613] Key type id_resolver registered [ 32.711393] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ *** ] A start job is running for Rebuild …amic Linker Cache (16s / no limit) [ ***] A start job is running for Rebuild …amic Linker Cache (17s / no limit) [ **] A start job is running for Rebuild …amic Linker Cache (18s / no limit) [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg150-server login: [ 218.427870] libcfs: loading out-of-tree module taints kernel. [ 218.698971] Key type ._llcrypt registered [ 218.718702] Key type .llcrypt registered [ 219.149020] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 285.810363] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 293.324470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 293.402679] alg: No test for adler32 (adler32-zlib) [ 296.914519] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 299.723371] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 302.185767] Key type lgssc registered [ 307.572686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 359.127312] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 486.838102] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 528.747905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 529.064513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.811237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 530.983630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 531.412773] Lustre: lustre-MDT0000: new disk, initializing [ 531.716339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.870911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 544.285841] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 584.168047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.773098] Lustre: 6585:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 584.951652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 585.005908] Lustre: Skipped 1 previous similar message [ 585.578975] Lustre: lustre-MDT0001: new disk, initializing [ 585.876327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 585.984818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 586.087757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 598.668657] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 611.181828] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 639.744593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 641.092538] Lustre: lustre-OST0000: new disk, initializing [ 641.127434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 641.502153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.835653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 647.876197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 648.042990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 658.557010] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 699.079936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 699.664727] Lustre: lustre-OST0001: new disk, initializing [ 699.692569] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 700.003438] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 708.355215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 708.408488] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 708.554642] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 720.484749] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 751.744777] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 768.002720] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 793.480656] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing check_logdir /tmp/testlogs/ [ 809.372030] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing yml_node [ 822.349361] Lustre: DEBUG MARKER: Client: 2.17.52.54 [ 829.666415] Lustre: DEBUG MARKER: MDS: 2.17.52.54 [ 836.788214] Lustre: DEBUG MARKER: OSS: 2.17.52.54 [ 841.884801] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Apr 23 21:45:47 EDT 2026 [ 895.898368] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 900.817007] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 933.858598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 933.865536] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.867083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 934.072939] Lustre: Skipped 3 previous similar messages [ 939.253464] Lustre: server umount lustre-MDT0000 complete [ 944.117759] LustreError: 6590:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 944.262870] LustreError: 6590:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 949.233820] LustreError: 10241:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.358447] LustreError: 10241:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 951.775474] LustreError: 6574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776995261 with bad export cookie 15623618185300549950 [ 951.789964] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 951.847197] LustreError: 6574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 954.337753] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.339745] LustreError: 8306:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 954.404263] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 954.404273] Lustre: Skipped 3 previous similar messages [ 954.526549] LustreError: 8306:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 956.390984] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 956.392668] LustreError: 10241:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.431206] Lustre: Skipped 2 previous similar messages [ 956.516223] LustreError: 10241:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 959.185987] Lustre: server umount lustre-MDT0001 complete [ 970.812958] Lustre: server umount lustre-OST0000 complete [ 981.792923] Lustre: server umount lustre-OST0001 complete [ 1002.143296] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 1035.750361] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 1148.504343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.867555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1150.016029] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1150.320854] Lustre: lustre-MDT0000: new disk, initializing [ 1150.666065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1150.770132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1163.152977] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1191.908505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1192.482271] Lustre: 15149:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1192.842280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1192.868111] Lustre: Skipped 1 previous similar message [ 1193.130007] Lustre: lustre-MDT0001: new disk, initializing [ 1193.434352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1193.696566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1193.768805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1207.393492] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1227.252644] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1237.892419] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1259.933815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1261.102891] Lustre: lustre-OST0000: new disk, initializing [ 1261.130217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1261.455220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.009892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1263.049130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1263.526515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1280.027903] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1299.122212] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1311.857736] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1312.602378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1326.547077] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1327.304007] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1335.268730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1335.304546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.342958] LustreError: Skipped 1 previous similar message [ 1335.400544] Lustre: Skipped 1 previous similar message [ 1335.404532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1340.397034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1340.456870] Lustre: Skipped 2 previous similar messages [ 1341.902694] Lustre: server umount lustre-OST0000 complete [ 1357.285371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1357.350552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.430252] Lustre: Skipped 1 previous similar message [ 1357.468304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1363.379136] Lustre: server umount lustre-MDT0000 complete [ 1366.008493] LustreError: 15162:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.078404] LustreError: 15162:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1375.206226] LustreError: 15142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776995684 with bad export cookie 15623618185300551483 [ 1375.223426] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1375.264411] LustreError: 15142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1376.436052] Lustre: server umount lustre-MDT0001 complete [ 1387.088013] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 21:54:53 (1776995693) [ 1406.126340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1408.547148] LustreError: 19034:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1408.658193] LustreError: 19034:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 1408.886960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1422.465322] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1429.476101] LustreError: 19035:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1429.568091] LustreError: 19035:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1450.070647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.809679] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1482.041494] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1494.725971] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1516.333094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1518.341520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1518.403944] Lustre: Skipped 1 previous similar message [ 1537.191068] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1556.417976] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1583.089245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1583.125303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1583.205378] Lustre: Skipped 1 previous similar message [ 1583.242247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1583.258975] Lustre: Skipped 2 previous similar messages [ 1587.729146] Lustre: server umount lustre-OST0000 complete [ 1599.970481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1600.023616] LustreError: Skipped 1 previous similar message [ 1600.068498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1600.103065] Lustre: Skipped 1 previous similar message [ 1605.436156] Lustre: server umount lustre-MDT0000 complete [ 1605.606699] LustreError: 19031:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1605.692791] LustreError: 19031:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 1617.441801] LustreError: 19014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776995927 with bad export cookie 15623618185300552365 [ 1617.483903] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1617.488149] LustreError: 19014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1618.918655] Lustre: server umount lustre-MDT0001 complete [ 1656.731899] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 1667.326618] Key type lgssc unregistered [ 1668.197349] LNet: 22278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.228259] LNetError: 22278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.285583] LNet: Removed LNI 192.168.201.150@tcp [ 1671.758364] Key type .llcrypt unregistered [ 1671.765911] Key type ._llcrypt unregistered [ 1751.187630] Key type ._llcrypt registered [ 1751.219819] Key type .llcrypt registered [ 1751.898167] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 1758.926014] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.621974] alg: No test for adler32 (adler32-zlib) [ 1761.440554] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 1762.597947] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 1764.711708] Key type lgssc registered [ 1769.070374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1795.691924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1795.940157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.209869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.340076] LustreError: 23700:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.414396] LustreError: 23700:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1811.251577] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1812.452043] LustreError: 23699:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1817.587301] LustreError: 23700:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1822.690626] LustreError: 23699:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1828.845703] LustreError: 23700:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1828.912908] LustreError: 23700:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1836.892544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1838.695230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1854.970549] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1871.795168] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1882.925129] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1903.486641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1905.309864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1919.654169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1925.444850] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1941.985573] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1990.634835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1990.675347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.755947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1991.651170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1991.656125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1995.303016] Lustre: server umount lustre-OST0000 complete [ 2007.014530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.112579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.133035] Lustre: Skipped 1 previous similar message [ 2012.129353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2012.180071] Lustre: Skipped 1 previous similar message [ 2013.596602] Lustre: server umount lustre-MDT0000 complete [ 2017.250313] LustreError: 23699:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.322082] LustreError: 23699:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2023.397965] LustreError: 24422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776996333 with bad export cookie 15142367571281802394 [ 2023.431212] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.472970] LustreError: 24422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2024.618235] Lustre: server umount lustre-MDT0001 complete [ 2060.645434] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 2069.962274] Key type lgssc unregistered [ 2071.088474] LNet: 27386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2071.131037] LNetError: 27386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2071.186120] LNet: Removed LNI 192.168.201.150@tcp [ 2073.783486] Key type .llcrypt unregistered [ 2073.806607] Key type ._llcrypt unregistered [ 2140.313259] Key type ._llcrypt registered [ 2140.333926] Key type .llcrypt registered [ 2140.777477] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 2145.962562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2146.385543] alg: No test for adler32 (adler32-zlib) [ 2148.031567] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 2148.897943] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 2150.887195] Key type lgssc registered [ 2154.537374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2178.026757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2178.110495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.575145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.845898] LustreError: 28796:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.984107] LustreError: 28796:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2195.160903] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2196.985957] LustreError: 28795:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.086502] LustreError: 28796:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.203233] LustreError: 28795:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.336508] LustreError: 28796:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.579853] LustreError: 28795:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.699744] LustreError: 28795:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2225.178193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.658891] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2227.113547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2241.650210] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2258.431010] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2270.819205] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2309.690610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2311.042099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2326.528991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2330.917140] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2350.301622] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2373.602056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2373.653236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.721609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.704696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2377.717564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.774804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.871761] Lustre: Skipped 1 previous similar message [ 2380.167645] Lustre: server umount lustre-OST0000 complete [ 2391.012887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2391.082614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.219620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2398.185981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2398.242280] Lustre: Skipped 3 previous similar messages [ 2405.344102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2406.101314] Lustre: server umount lustre-MDT0000 complete [ 2407.403654] LustreError: 29517:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.454560] LustreError: 29517:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2419.584307] LustreError: 28777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776996729 with bad export cookie 4160341890164706980 [ 2419.627583] LustreError: 28777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2419.628426] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2421.128554] Lustre: server umount lustre-MDT0001 complete [ 2456.747796] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 2464.940563] Key type lgssc unregistered [ 2466.050664] LNet: 32389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2466.094713] LNetError: 32389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2466.145320] LNet: Removed LNI 192.168.201.150@tcp [ 2468.550589] Key type .llcrypt unregistered [ 2468.566881] Key type ._llcrypt unregistered [ 2542.091300] Key type ._llcrypt registered [ 2542.114212] Key type .llcrypt registered [ 2542.442500] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 2548.394279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2548.464301] alg: No test for adler32 (adler32-zlib) [ 2550.572239] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 2551.282892] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 2553.369591] Key type lgssc registered [ 2556.859036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2581.320334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2581.438057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.765203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2594.272617] LustreError: 33797:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.352371] LustreError: 33797:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2598.659365] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2599.398435] LustreError: 33798:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.516180] LustreError: 33797:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.646595] LustreError: 33798:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.764370] LustreError: 33797:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.775916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.000567] LustreError: 33798:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.079463] LustreError: 33798:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2625.853133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2638.967968] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2655.366865] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2667.105978] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2687.467572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2688.489162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2705.023932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2707.322687] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2727.271877] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2756.078880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.218538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2761.215566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2761.256496] Lustre: Skipped 1 previous similar message [ 2766.320573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2766.372621] Lustre: Skipped 2 previous similar messages [ 2767.327701] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2768.543370] Lustre: server umount lustre-OST0000 complete [ 2784.739112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2784.794618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.890153] Lustre: Skipped 1 previous similar message [ 2784.945563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2786.787334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2789.829011] Lustre: server umount lustre-MDT0000 complete [ 2791.908450] LustreError: 33797:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.028075] LustreError: 33797:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2802.708816] LustreError: 33778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776997112 with bad export cookie 14912529112951767814 [ 2802.744023] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2802.781339] LustreError: 33778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2804.273185] Lustre: server umount lustre-MDT0001 complete [ 2840.521297] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 2849.488913] Key type lgssc unregistered [ 2850.453229] LNet: 37040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2850.485147] LNetError: 37040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2850.554415] LNet: Removed LNI 192.168.201.150@tcp [ 2853.527615] Key type .llcrypt unregistered [ 2853.533234] Key type ._llcrypt unregistered [ 2924.476574] Key type ._llcrypt registered [ 2924.490223] Key type .llcrypt registered [ 2924.998113] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 2928.909161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2930.362261] alg: No test for adler32 (adler32-zlib) [ 2932.884586] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 2934.104333] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 2936.239405] Key type lgssc registered [ 2939.466013] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2964.681743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2964.754837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2967.829709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.273221] LustreError: 38448:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2978.359741] LustreError: 38448:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2981.758515] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2983.416885] LustreError: 38447:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2988.535020] LustreError: 38448:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2993.636129] LustreError: 38447:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.774824] LustreError: 38448:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.993718] LustreError: 38447:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.092753] LustreError: 38447:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3009.863667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.635473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3028.201167] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3044.431676] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3056.157932] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3077.777800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3079.410353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3088.681601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3098.660671] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3118.048525] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3133.922757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3133.960474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.043612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3134.949319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3134.964690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3134.984799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.081897] Lustre: Skipped 1 previous similar message [ 3137.159415] Lustre: lustre-OST0000: Not available for connect from 192.168.201.50@tcp (stopping) [ 3139.212352] Lustre: server umount lustre-OST0000 complete [ 3180.001769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.036943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3180.057792] Lustre: Skipped 1 previous similar message [ 3180.108060] Lustre: Skipped 1 previous similar message [ 3185.001029] Lustre: server umount lustre-MDT0000 complete [ 3185.123816] LustreError: 38443:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.186520] LustreError: 38443:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3195.449097] LustreError: 39171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776997505 with bad export cookie 7523185554075920643 [ 3195.459356] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3195.484852] LustreError: 39171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3202.896476] Lustre: server umount lustre-MDT0001 complete [ 3237.231768] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 3247.660896] Key type lgssc unregistered [ 3248.818066] LNet: 41851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3248.883613] LNetError: 41851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3248.957504] LNet: Removed LNI 192.168.201.150@tcp [ 3251.572319] Key type .llcrypt unregistered [ 3251.600656] Key type ._llcrypt unregistered [ 3324.667180] Key type ._llcrypt registered [ 3324.678680] Key type .llcrypt registered [ 3325.146434] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 3328.829353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3329.137171] alg: No test for adler32 (adler32-zlib) [ 3332.423511] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 3333.158237] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 3335.063417] Key type lgssc registered [ 3338.859435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3367.504242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3367.596650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.816190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.221607] LustreError: 43253:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3381.298463] LustreError: 43253:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3382.242678] LustreError: 43254:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.004831] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3387.373122] LustreError: 43253:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.497131] LustreError: 43254:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3396.590160] LustreError: 43253:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.823356] LustreError: 43254:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.954031] LustreError: 43254:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3410.605358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.898202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3425.168909] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3442.571672] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3454.701570] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3476.687975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3478.960267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.333360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3498.423008] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3514.916401] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3529.704816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3529.757628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.842349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3533.299090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3533.327487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.363715] Lustre: Skipped 1 previous similar message [ 3533.418148] Lustre: Skipped 1 previous similar message [ 3536.433008] Lustre: server umount lustre-MDT0000 complete [ 3538.410066] LustreError: 43253:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.475982] LustreError: 43253:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3548.205854] LustreError: 43234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776997857 with bad export cookie 14369323606363575970 [ 3548.237621] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3548.264763] LustreError: 43234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3548.653285] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.689042] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3548.710523] Lustre: Skipped 1 previous similar message [ 3553.771323] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3553.820765] Lustre: Skipped 1 previous similar message [ 3555.468752] Lustre: server umount lustre-MDT0001 complete [ 3579.831860] Lustre: server umount lustre-OST0000 complete [ 3628.875433] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 3638.570831] Key type lgssc unregistered [ 3639.455089] LNet: 46807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3639.508127] LNetError: 46807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3639.590540] LNet: Removed LNI 192.168.201.150@tcp [ 3642.482664] Key type .llcrypt unregistered [ 3642.502218] Key type ._llcrypt unregistered [ 3710.235458] Key type ._llcrypt registered [ 3710.259108] Key type .llcrypt registered [ 3710.644723] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 3715.906031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3716.068401] alg: No test for adler32 (adler32-zlib) [ 3717.782306] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 3718.585372] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 3720.711262] Key type lgssc registered [ 3725.142005] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3748.110434] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3748.239681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3764.831618] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3770.015344] LustreError: 48185:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.150@tcp: failed processing log, type 4: rc = -110 [ 3796.959403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.845985] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3832.028390] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4003.382666] Lustre: server umount lustre-OST0000 complete [ 4061.389443] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 4070.298525] Key type lgssc unregistered [ 4071.382436] LNet: 49763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4071.418747] LNetError: 49763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4071.491482] LNet: Removed LNI 192.168.201.150@tcp [ 4074.190580] Key type .llcrypt unregistered [ 4074.206594] Key type ._llcrypt unregistered [ 4146.913647] Key type ._llcrypt registered [ 4146.943094] Key type .llcrypt registered [ 4147.460046] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 4152.774647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4152.839455] alg: No test for adler32 (adler32-zlib) [ 4154.299494] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 4154.859732] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 4156.879371] Key type lgssc registered [ 4161.068044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4184.587797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4184.687978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.726875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4197.855916] LustreError: 51170:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4197.922018] LustreError: 51170:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4198.900780] LustreError: 51169:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4201.202018] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 4204.008267] LustreError: 51170:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4209.126024] LustreError: 51169:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4214.264233] LustreError: 51170:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4224.492218] LustreError: 51169:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4224.609078] LustreError: 51169:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4229.231499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4231.341614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4244.544008] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 4263.638882] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4276.801768] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4296.247975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4297.997374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4309.369518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4317.192621] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 4333.980325] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4342.241310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4342.271149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.339422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4345.330135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4345.336161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.491477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4348.673663] Lustre: server umount lustre-OST0000 complete [ 4359.527482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4359.609068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4359.641025] Lustre: Skipped 1 previous similar message [ 4364.263112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4364.305524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4364.364932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4366.356586] Lustre: server umount lustre-MDT0000 complete [ 4369.903486] LustreError: 51169:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4369.926811] LustreError: 51169:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4377.611222] LustreError: 51893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776998687 with bad export cookie 8555510952266983703 [ 4377.612832] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4377.692574] LustreError: 51893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4378.874657] Lustre: server umount lustre-MDT0001 complete [ 4412.764673] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 4422.061951] Key type lgssc unregistered [ 4423.042942] LNet: 54387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4423.075287] LNetError: 54387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4423.133964] LNet: Removed LNI 192.168.201.150@tcp [ 4426.162553] Key type .llcrypt unregistered [ 4426.181675] Key type ._llcrypt unregistered [ 4497.452589] Key type ._llcrypt registered [ 4497.485993] Key type .llcrypt registered [ 4497.770646] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 4502.482525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4502.693413] alg: No test for adler32 (adler32-zlib) [ 4504.762470] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 4505.468575] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 4507.431664] Key type lgssc registered [ 4511.167619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4535.238438] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4535.346389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4551.521549] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4556.703343] LustreError: 55769:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.150@tcp: failed processing log, type 4: rc = -110 [ 4583.711269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4604.000282] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 4622.073129] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4647.987527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4650.566749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4650.861601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 4655.593495] LustreError: 55795:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.653280] LustreError: 55795:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4660.737031] LustreError: 55794:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.465708] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 4665.831695] LustreError: 55795:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4668.913550] LustreError: 55794:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4669.024439] LustreError: 55794:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4674.060054] LustreError: 55795:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4674.162915] LustreError: 55795:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4684.275692] LustreError: 56710:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4684.418431] LustreError: 56710:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4696.871737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.853047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4716.102147] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 4741.428576] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4753.888731] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4760.037738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4760.091187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4760.180847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4765.672323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4765.704641] Lustre: Skipped 2 previous similar messages [ 4766.092515] Lustre: server umount lustre-OST0000 complete [ 4816.370558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4816.382673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4816.432810] Lustre: Skipped 1 previous similar message [ 4816.454733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4819.727244] Lustre: server umount lustre-MDT0000 complete [ 4823.013867] LustreError: 56709:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.120157] LustreError: 56709:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 4830.044565] LustreError: 55775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776999139 with bad export cookie 17999079817990590945 [ 4830.081341] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4830.084316] LustreError: 55775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4837.524626] Lustre: server umount lustre-MDT0001 complete [ 4873.059823] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 4882.086377] Key type lgssc unregistered [ 4883.034831] LNet: 59215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4883.065787] LNetError: 59215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4883.125730] LNet: Removed LNI 192.168.201.150@tcp [ 4886.048217] Key type .llcrypt unregistered [ 4886.064254] Key type ._llcrypt unregistered [ 4952.079210] Key type ._llcrypt registered [ 4952.093064] Key type .llcrypt registered [ 4952.453558] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 4959.023004] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4959.078907] alg: No test for adler32 (adler32-zlib) [ 4960.758326] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 4961.638703] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 4963.795979] Key type lgssc registered [ 4967.418732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4990.195066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4990.377893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4993.514107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5003.766183] LustreError: 60621:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.878852] LustreError: 60621:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5006.078184] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 5008.884113] LustreError: 60622:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5014.005156] LustreError: 60621:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5019.122635] LustreError: 60622:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5025.254469] LustreError: 60621:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5025.414764] LustreError: 60621:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5035.491197] LustreError: 60622:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5035.539236] LustreError: 60622:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5035.916052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5037.445656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5050.678020] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 5067.545958] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5079.130918] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5100.681708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5102.260723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.712586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 5121.813751] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 5136.032565] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5168.097719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5168.134315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5168.195176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5171.173165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5171.184428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5171.252086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5173.406225] Lustre: server umount lustre-OST0000 complete [ 5185.536591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5185.586243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5185.683409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5185.717592] Lustre: Skipped 1 previous similar message [ 5191.653903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.713279] Lustre: Skipped 3 previous similar messages [ 5192.602823] Lustre: server umount lustre-MDT0000 complete [ 5196.791746] LustreError: 60621:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5196.898750] LustreError: 60621:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5203.763594] LustreError: 60603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776999513 with bad export cookie 4206068883078978277 [ 5203.789831] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5203.804460] LustreError: 60603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5205.922888] Lustre: server umount lustre-MDT0001 complete [ 5240.170993] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 5247.964062] Key type lgssc unregistered [ 5248.742136] LNet: 63862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5248.788935] LNetError: 63862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5248.821634] LNet: Removed LNI 192.168.201.150@tcp [ 5251.333857] Key type .llcrypt unregistered [ 5251.355051] Key type ._llcrypt unregistered [ 5327.606599] Key type ._llcrypt registered [ 5327.633635] Key type .llcrypt registered [ 5328.016278] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 5332.330384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5332.495237] alg: No test for adler32 (adler32-zlib) [ 5334.615046] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 5335.519452] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 5337.671743] Key type lgssc registered [ 5341.665005] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5370.188105] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 23:01:14 (1776999674) [ 5391.285234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5391.391637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.983510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5404.137171] LustreError: 65582:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.290300] LustreError: 65582:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5406.976015] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 5409.261519] LustreError: 65581:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5414.369950] LustreError: 65582:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5418.506761] LustreError: 65581:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5423.598312] LustreError: 65582:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5433.828902] LustreError: 65581:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5433.919921] LustreError: 65581:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5435.145958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5436.932834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5447.251040] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 5463.604387] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5474.276873] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5491.773950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5493.104666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5509.132772] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 5513.704057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 5525.280659] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5538.327725] Lustre: Failing over lustre-MDT0000 [ 5538.491351] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.50@tcp (stopping) [ 5539.330678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5539.353827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5539.394212] Lustre: Skipped 1 previous similar message [ 5539.431867] LustreError: 67746:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5539.574759] LustreError: 67746:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5539.677181] Lustre: server umount lustre-MDT0000 complete [ 5555.616701] Lustre: 64632:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776999849/real 1776999849] req@00000000ae2a764c x1863319179797504/t0(0) o400->MGC192.168.201.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776999865 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5555.706784] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5565.529146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5574.707970] LustreError: 67746:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.805278] LustreError: 67746:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 27 previous similar messages [ 5581.315765] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8fc9d07b860681fb [ 5581.397890] Lustre: MGC192.168.201.150@tcp: Connection restored to 0@lo (at 0@lo) [ 5581.828430] Lustre: *** cfs_fail_loc=135, val=0*** [ 5582.324340] LustreError: 68348:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5582.372132] LustreError: 68348:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.150@tcp: cfg command failed: rc = -2 [ 5582.417193] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5582.417193] [ 5582.498248] LustreError: MGC192.168.201.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5582.602299] LustreError: 68333:0:(tgt_mount.c:1838:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5582.666088] LustreError: 68333:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -2 [ 5582.730503] LustreError: 68333:0:(tgt_mount.c:2082:server_put_super()) no obd lustre-MDT0000 [ 5582.785751] LustreError: 68333:0:(tgt_mount.c:114:server_deregister_mount()) lustre-MDT0000 not registered [ 5582.938389] Lustre: server umount lustre-MDT0000 complete [ 5582.951069] LustreError: 68333:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5602.292141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5602.789082] Lustre: 64632:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776999896/real 1776999896] req@00000000ae7e0e58 x1863319179807872/t0(0) o400->MGC192.168.201.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776999912 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5602.919540] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5602.979946] LustreError: 65574:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 5613.323727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5613.530345] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5616.175356] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5618.685078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5618.778809] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5619.002917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 5625.590931] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 5649.401401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5649.416224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5649.442991] Lustre: Skipped 2 previous similar messages [ 5654.503977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5654.530147] Lustre: Skipped 2 previous similar messages [ 5656.173701] Lustre: server umount lustre-OST0000 complete [ 5669.860550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5669.869338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5669.908552] Lustre: Skipped 1 previous similar message [ 5669.976993] Lustre: Skipped 1 previous similar message [ 5674.307816] Lustre: server umount lustre-MDT0000 complete [ 5674.986674] LustreError: 65582:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5675.089294] LustreError: 65582:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 36 previous similar messages [ 5685.435689] LustreError: 65561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776999995 with bad export cookie 10361041646690665123 [ 5685.451769] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5685.492781] LustreError: 65561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5686.846647] Lustre: server umount lustre-MDT0001 complete [ 5724.675455] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 5734.048686] Key type lgssc unregistered [ 5735.281720] LNet: 70153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5735.345406] LNetError: 70153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5735.408699] LNet: Removed LNI 192.168.201.150@tcp [ 5738.263754] Key type .llcrypt unregistered [ 5738.269294] Key type ._llcrypt unregistered [ 5805.975595] Key type ._llcrypt registered [ 5805.987754] Key type .llcrypt registered [ 5806.377742] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 5811.598863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5811.803866] alg: No test for adler32 (adler32-zlib) [ 5813.956959] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 5814.830147] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 5816.871918] Key type lgssc registered [ 5820.651019] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5843.897913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5844.003960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5846.730919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5857.256913] LustreError: 71566:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5857.333591] LustreError: 71566:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5860.933469] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 5862.375072] LustreError: 71565:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.494349] LustreError: 71566:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.600984] LustreError: 71565:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5876.708870] LustreError: 71566:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5886.482737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5886.949219] LustreError: 71565:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.045890] LustreError: 71565:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5887.920629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5898.533651] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 5914.379851] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5927.605519] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5947.592089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5949.514350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.346823] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 5967.029542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 5980.133267] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5992.512799] Lustre: Failing over lustre-MDT0000 [ 5992.953076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.958498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5993.027326] Lustre: Skipped 1 previous similar message [ 5993.046419] Lustre: Skipped 1 previous similar message [ 5993.370377] Lustre: server umount lustre-MDT0000 complete [ 5994.140507] LustreError: 73453:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6011.295500] Lustre: 70873:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777000305/real 1777000305] req@00000000123d13bc x1863319682066816/t0(0) o400->MGC192.168.201.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777000321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6011.448365] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6018.786982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6021.535875] LustreError: 74317:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 6021.565265] LustreError: 74317:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@000000004091e90b x1863319682070272/t0(0) o250->MGC192.168.201.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1777000331 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6021.742817] LustreError: 74317:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 6021.892047] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x86ed21b85144a30d [ 6021.941084] Lustre: MGC192.168.201.150@tcp: Connection restored to 0@lo (at 0@lo) [ 6022.436467] Lustre: *** cfs_fail_loc=a05, val=0*** [ 6022.852352] LustreError: 74330:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 6022.888920] LustreError: 74330:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.150@tcp: cfg command failed: rc = -9 [ 6022.939753] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6022.939753] [ 6022.992317] LustreError: MGC192.168.201.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6023.113970] LustreError: 74317:0:(tgt_mount.c:1838:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 6023.160703] LustreError: 74317:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -9 [ 6023.208465] LustreError: 74317:0:(tgt_mount.c:2082:server_put_super()) no obd lustre-MDT0000 [ 6023.236294] LustreError: 74317:0:(tgt_mount.c:114:server_deregister_mount()) lustre-MDT0000 not registered [ 6023.392881] Lustre: server umount lustre-MDT0000 complete [ 6023.415485] LustreError: 74317:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 6027.237975] LustreError: 71566:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6027.320579] LustreError: 71566:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 28 previous similar messages [ 6042.633025] Lustre: 70876:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777000336/real 1777000336] req@00000000a7fd53e0 x1863319682073472/t0(0) o400->MGC192.168.201.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777000352 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6042.768150] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6042.794566] LustreError: 71558:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 6042.851283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6053.436109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6053.697210] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6057.540806] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6058.523742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6058.608868] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6058.824857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 6065.989640] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6089.704350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6089.724693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6089.760986] Lustre: Skipped 1 previous similar message [ 6089.779024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6089.819724] Lustre: Skipped 1 previous similar message [ 6091.233501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6091.288613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.341687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6094.306735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6094.329271] Lustre: Skipped 1 previous similar message [ 6095.758256] Lustre: server umount lustre-OST0000 complete [ 6109.684375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6109.689050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6109.743324] Lustre: Skipped 1 previous similar message [ 6112.832945] Lustre: server umount lustre-MDT0000 complete [ 6114.795740] LustreError: 71565:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6114.864125] LustreError: 71565:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 22 previous similar messages [ 6122.930656] LustreError: 71546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777000432 with bad export cookie 9722464246083920821 [ 6122.956607] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6122.998228] LustreError: 71546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6124.072166] Lustre: server umount lustre-MDT0001 complete [ 6155.446581] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 6164.062600] Key type lgssc unregistered [ 6165.055427] LNet: 76136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6165.106573] LNetError: 76136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6165.160868] LNet: Removed LNI 192.168.201.150@tcp [ 6168.110697] Key type .llcrypt unregistered [ 6168.113980] Key type ._llcrypt unregistered [ 6205.538734] Key type ._llcrypt registered [ 6205.554899] Key type .llcrypt registered [ 6205.941191] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 6206.970335] Key type .llcrypt unregistered [ 6206.983356] Key type ._llcrypt unregistered [ 6268.580823] Key type ._llcrypt registered [ 6268.594319] Key type .llcrypt registered [ 6268.859320] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 6271.564131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6271.712570] alg: No test for adler32 (adler32-zlib) [ 6274.667897] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 6275.367875] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 6277.375251] Key type lgssc registered [ 6280.206379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6302.979685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6303.057514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6306.151855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6315.510339] LustreError: 78022:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6315.585597] LustreError: 78022:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6318.496737] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6320.620384] LustreError: 78021:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.743692] LustreError: 78022:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.861554] LustreError: 78021:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6335.999244] LustreError: 78022:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6344.738974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.489516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6358.408943] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6374.100799] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6385.684038] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6406.238896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6408.286939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6425.145818] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6425.745423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 6440.189373] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6467.046222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6467.057717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.059955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6467.098190] LustreError: Skipped 1 previous similar message [ 6467.168782] Lustre: Skipped 1 previous similar message [ 6471.854615] Lustre: server umount lustre-OST0000 complete [ 6484.456716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6484.492445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6484.537734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6484.560952] Lustre: Skipped 1 previous similar message [ 6486.633549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6486.652170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.726697] Lustre: server umount lustre-MDT0000 complete [ 6492.132687] LustreError: 78017:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6492.239575] LustreError: 78017:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6500.264247] LustreError: 78003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777000810 with bad export cookie 2542589163188220300 [ 6500.293521] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6500.322465] LustreError: 78003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6501.171877] Lustre: server umount lustre-MDT0001 complete [ 6531.576758] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 6539.796791] Key type lgssc unregistered [ 6540.918045] LNet: 81260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6540.982849] LNetError: 81260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6541.042996] LNet: Removed LNI 192.168.201.150@tcp [ 6544.197114] Key type .llcrypt unregistered [ 6544.211739] Key type ._llcrypt unregistered [ 6611.406533] Key type ._llcrypt registered [ 6611.418585] Key type .llcrypt registered [ 6611.809776] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 6616.414609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6616.501666] alg: No test for adler32 (adler32-zlib) [ 6618.401654] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 6619.256049] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 6621.293363] Key type lgssc registered [ 6624.597482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6649.090055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6649.168793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.411509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6661.615090] LustreError: 82669:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6661.752136] LustreError: 82669:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6665.041938] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6666.722571] LustreError: 82670:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6671.871772] LustreError: 82669:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6676.979696] LustreError: 82670:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.116411] LustreError: 82669:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.192057] LustreError: 82669:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6689.587680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.161413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6704.803325] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6742.302057] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6757.361706] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6781.594873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6783.319250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6797.759920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 6801.487920] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6818.369664] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6833.643596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6833.647008] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6833.706199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6833.739561] Lustre: Skipped 1 previous similar message [ 6838.861743] Lustre: server umount lustre-OST0000 complete [ 6850.018913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6850.050660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6850.137580] Lustre: Skipped 1 previous similar message [ 6850.172394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6853.293101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6853.294365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6853.390928] Lustre: Skipped 1 previous similar message [ 6856.170887] Lustre: server umount lustre-MDT0000 complete [ 6858.725077] LustreError: 82670:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6858.788290] LustreError: 82670:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6869.070138] LustreError: 82649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777001178 with bad export cookie 9029687211848405174 [ 6869.090693] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6869.131839] LustreError: 82649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6870.868715] Lustre: server umount lustre-MDT0001 complete [ 6906.233302] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 6914.973932] Key type lgssc unregistered [ 6915.926780] LNet: 85919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6915.965040] LNetError: 85919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6916.031266] LNet: Removed LNI 192.168.201.150@tcp [ 6918.726678] Key type .llcrypt unregistered [ 6918.743139] Key type ._llcrypt unregistered [ 6990.960126] Key type ._llcrypt registered [ 6990.973229] Key type .llcrypt registered [ 6991.475636] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 6996.114505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6996.193019] alg: No test for adler32 (adler32-zlib) [ 6997.970167] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 6998.823329] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 7000.947251] Key type lgssc registered [ 7004.703449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7027.514002] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7027.625964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7030.695730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7040.995566] LustreError: 87328:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7041.093119] LustreError: 87328:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7044.057920] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7046.132259] LustreError: 87327:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7051.244924] LustreError: 87328:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7055.342716] LustreError: 87327:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7060.453537] LustreError: 87328:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.970336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7070.699223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7083.579816] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7100.917602] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7112.805513] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7134.661462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7136.503991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7151.025160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 7154.011318] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7171.807160] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7212.540355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7212.554324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7212.611851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7212.642602] Lustre: Skipped 1 previous similar message [ 7212.719639] Lustre: Skipped 1 previous similar message [ 7217.651088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7217.726626] Lustre: Skipped 1 previous similar message [ 7222.770410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7222.806569] Lustre: Skipped 1 previous similar message [ 7223.265065] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7223.961850] Lustre: server umount lustre-OST0000 complete [ 7238.112811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7238.117466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7238.152208] Lustre: Skipped 1 previous similar message [ 7238.175790] Lustre: Skipped 1 previous similar message [ 7242.517828] Lustre: server umount lustre-MDT0000 complete [ 7243.260494] LustreError: 88122:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7243.354722] LustreError: 88122:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7253.206614] LustreError: 87308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777001562 with bad export cookie 612137610179846189 [ 7253.256680] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7253.313547] LustreError: 87308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7254.583542] Lustre: server umount lustre-MDT0001 complete [ 7291.851908] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 7300.373170] Key type lgssc unregistered [ 7301.338184] LNet: 90575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7301.381593] LNetError: 90575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7301.433766] LNet: Removed LNI 192.168.201.150@tcp [ 7304.352577] Key type .llcrypt unregistered [ 7304.357728] Key type ._llcrypt unregistered [ 7372.846718] Key type ._llcrypt registered [ 7372.872661] Key type .llcrypt registered [ 7373.421544] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 7379.332768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7379.382863] alg: No test for adler32 (adler32-zlib) [ 7381.285593] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 7382.178607] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 7384.191316] Key type lgssc registered [ 7388.190047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7411.708692] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7411.803115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7427.872408] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7433.055239] LustreError: 91958:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.150@tcp: failed processing log, type 4: rc = -110 [ 7459.999616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7477.924744] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7493.995958] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7510.757123] Lustre: server umount lustre-OST0000 complete [ 7535.615984] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 23:37:22 (1777001842) [ 7556.836555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7558.811191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7564.267509] LustreError: 93493:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7564.330128] LustreError: 93493:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7568.362116] LustreError: 93494:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7570.675795] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7573.474215] LustreError: 93493:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.593414] LustreError: 93494:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7583.720138] LustreError: 93493:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7593.960006] LustreError: 93494:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.041803] LustreError: 93494:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7595.775659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7597.523895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7609.700322] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7626.083356] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7637.299463] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7656.607192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7657.432075] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7675.474673] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7678.981350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 7691.453448] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7717.866250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7717.893215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7717.967933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7719.333226] Lustre: server umount lustre-OST0000 complete [ 7734.984631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.013232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7735.052728] Lustre: Skipped 1 previous similar message [ 7735.163946] Lustre: Skipped 1 previous similar message [ 7740.390118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7740.424335] Lustre: Skipped 1 previous similar message [ 7741.642116] Lustre: server umount lustre-MDT0000 complete [ 7745.506102] LustreError: 93490:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.593556] LustreError: 93490:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7752.955062] LustreError: 93473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777002062 with bad export cookie 1071531887205348902 [ 7752.995595] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7753.030197] LustreError: 93473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7754.089025] Lustre: server umount lustre-MDT0001 complete [ 7787.021896] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 7796.268268] Key type lgssc unregistered [ 7796.870368] LNet: 96871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7796.896087] LNetError: 96871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7796.987080] LNet: Removed LNI 192.168.201.150@tcp [ 7799.656711] Key type .llcrypt unregistered [ 7799.677562] Key type ._llcrypt unregistered [ 7913.849992] Key type ._llcrypt registered [ 7913.858093] Key type .llcrypt registered [ 7914.302882] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 7919.807803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7919.896454] alg: No test for adler32 (adler32-zlib) [ 7921.903979] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 7922.678240] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 7924.610619] Key type lgssc registered [ 7927.800593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7948.463027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7948.572066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7951.877782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7962.080715] LustreError: 99055:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7962.140375] LustreError: 99055:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7963.125806] LustreError: 99056:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7963.783103] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7968.249325] LustreError: 99055:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7973.404492] LustreError: 99056:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7977.509126] LustreError: 99055:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7987.686442] LustreError: 99056:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7987.760789] LustreError: 99056:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7988.142561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7989.709819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8001.051469] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8017.673980] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8028.300985] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8046.088694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8047.541106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8048.607698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 8063.091622] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8077.665436] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8099.812219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8099.863959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8099.908222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8100.844353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8100.863277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.869188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8100.929809] Lustre: Skipped 1 previous similar message [ 8105.066298] Lustre: server umount lustre-OST0000 complete [ 8116.199351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8116.204679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8116.289816] Lustre: Skipped 1 previous similar message [ 8116.359103] Lustre: Skipped 1 previous similar message [ 8120.357766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8120.379449] Lustre: Skipped 1 previous similar message [ 8121.518267] Lustre: server umount lustre-MDT0000 complete [ 8125.415645] LustreError: 99050:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8125.497626] LustreError: 99050:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8131.333275] LustreError: 99037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777002441 with bad export cookie 14778469768393168109 [ 8131.349966] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8131.392578] LustreError: 99037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8132.668401] Lustre: server umount lustre-MDT0001 complete [ 8163.939833] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 8171.466732] Key type lgssc unregistered [ 8172.231702] LNet: 102289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8172.275184] LNetError: 102289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8172.357596] LNet: Removed LNI 192.168.201.150@tcp [ 8174.740891] Key type .llcrypt unregistered [ 8174.754257] Key type ._llcrypt unregistered [ 8221.775212] Key type ._llcrypt registered [ 8221.793464] Key type .llcrypt registered [ 8222.120130] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 8225.872316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8225.906812] alg: No test for adler32 (adler32-zlib) [ 8227.481986] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 8228.339226] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 8230.495275] Key type lgssc registered [ 8233.796729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8255.426062] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8255.484338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8271.711484] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8276.895410] LustreError: 103527:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.150@tcp: failed processing log, type 4: rc = -110 [ 8303.775539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8321.342885] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8336.131615] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8353.179876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8354.754790] LustreError: MGC192.168.201.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 8354.785746] LustreError: 104442:0:(tgt_mount.c:1838:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 8354.828090] LustreError: 104442:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -2 [ 8354.856729] LustreError: 104442:0:(tgt_mount.c:2082:server_put_super()) no obd lustre-MDT0000 [ 8355.204157] Lustre: server umount lustre-MDT0000 complete [ 8355.222829] LustreError: 104442:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8372.703322] Lustre: 102863:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777002666/real 1777002666] req@00000000b7dc7fdb x1863322213289728/t0(0) o400->MGC192.168.201.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777002682 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8372.794300] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8372.815307] LustreError: 103539:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 8393.515873] Lustre: server umount lustre-OST0000 complete [ 8418.804403] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 8446.161744] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 8538.987979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8539.721325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8539.852362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8540.091672] Lustre: lustre-MDT0000: new disk, initializing [ 8540.269128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.378814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8552.396372] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8576.602559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8576.965952] Lustre: 107239:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8577.160903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8577.186299] Lustre: Skipped 1 previous similar message [ 8577.510231] Lustre: lustre-MDT0001: new disk, initializing [ 8577.759716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8577.859073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8577.889324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8587.905707] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8603.575957] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8612.218817] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8628.159382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8628.959009] Lustre: lustre-OST0000: new disk, initializing [ 8628.978271] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8629.312009] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 8635.723743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8635.776142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8635.995675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8644.988889] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8662.352922] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8671.806701] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8672.423324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8682.602757] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8683.308425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8692.199795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8692.245152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8692.290068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8692.331911] Lustre: Skipped 1 previous similar message [ 8694.737617] Lustre: server umount lustre-OST0000 complete [ 8703.749854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.757503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8703.823414] Lustre: Skipped 2 previous similar messages [ 8703.873412] Lustre: Skipped 1 previous similar message [ 8709.123613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8709.154938] Lustre: Skipped 1 previous similar message [ 8713.252548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8713.288255] Lustre: Skipped 1 previous similar message [ 8718.303684] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8718.310372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8718.407449] Lustre: Skipped 1 previous similar message [ 8719.344018] Lustre: server umount lustre-MDT0000 complete [ 8728.550634] LustreError: 107253:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8728.634666] LustreError: 107253:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 8728.703321] LustreError: 107233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777003038 with bad export cookie 2969062676365151321 [ 8728.710734] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8728.730458] LustreError: 107233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8730.056818] Lustre: server umount lustre-MDT0001 complete [ 8751.630854] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 23:57:37 (1777003057) [ 8759.330265] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 8797.251422] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 8824.575518] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 8916.082371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8917.168564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8917.297283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8917.580522] Lustre: lustre-MDT0000: new disk, initializing [ 8917.830136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8917.899135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8929.972477] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8953.827244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8954.154566] Lustre: 113759:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8954.312200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8954.327449] Lustre: Skipped 1 previous similar message [ 8954.593762] Lustre: lustre-MDT0001: new disk, initializing [ 8954.904789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8955.054139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8955.103993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8964.074006] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8978.787830] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8988.835066] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9006.181914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9007.006983] Lustre: lustre-OST0000: new disk, initializing [ 9007.022293] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9007.260423] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 9012.871533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9012.906956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9013.109085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9019.995443] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9033.557858] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9042.949266] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9043.514930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9054.528223] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9054.975647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9059.301482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9059.306949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9059.359273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9059.381473] LustreError: Skipped 1 previous similar message [ 9059.403825] Lustre: Skipped 1 previous similar message [ 9065.696779] Lustre: server umount lustre-OST0000 complete [ 9074.658138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9074.711522] Lustre: Skipped 1 previous similar message [ 9079.783053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9079.801684] Lustre: Skipped 5 previous similar messages [ 9080.047224] Lustre: server umount lustre-MDT0000 complete [ 9084.906996] LustreError: 114734:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9084.980952] LustreError: 114734:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 9088.437583] LustreError: 113751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777003398 with bad export cookie 2969062676365152203 [ 9088.439592] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9088.469812] LustreError: 113751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9089.503031] Lustre: server umount lustre-MDT0001 complete [ 9113.572503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9115.123415] LustreError: 117595:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9115.369513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9120.738374] LustreError: 117596:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9124.842894] LustreError: 117595:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9126.374112] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9135.105231] LustreError: 117596:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9135.157698] LustreError: 117596:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 9145.242997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9146.744282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9156.663884] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9169.100380] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9178.390202] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9193.569803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9194.497436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9207.729145] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9221.472947] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9232.828746] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 9242.599450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9242.656844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9242.692366] Lustre: Skipped 1 previous similar message [ 9242.724433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9245.676721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9247.921545] Lustre: server umount lustre-OST0000 complete [ 9257.162653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9257.211388] Lustre: Skipped 1 previous similar message [ 9263.941817] Lustre: server umount lustre-MDT0000 complete [ 9267.683700] LustreError: 118631:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9267.746025] LustreError: 118631:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 9272.242560] LustreError: 117577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777003581 with bad export cookie 2969062676365153085 [ 9272.259010] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9272.313868] LustreError: 117577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9273.126525] Lustre: server umount lustre-MDT0001 complete [ 9299.058902] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 9305.889172] Key type lgssc unregistered [ 9306.537857] LNet: 120905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9306.575435] LNetError: 120905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9306.629228] LNet: Removed LNI 192.168.201.150@tcp [ 9308.370432] Key type .llcrypt unregistered [ 9308.376257] Key type ._llcrypt unregistered [ 9349.531508] Key type ._llcrypt registered [ 9349.537568] Key type .llcrypt registered [ 9349.882772] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 9381.845659] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 9385.409579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9385.446760] alg: No test for adler32 (adler32-zlib) [ 9386.930253] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [ 9387.649381] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 9389.543605] Key type lgssc registered [ 9391.886964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9463.579096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9463.647678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9465.243487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9465.305553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9465.515198] Lustre: lustre-MDT0000: new disk, initializing [ 9465.836908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9465.898869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9474.811391] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9497.274459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9497.680598] Lustre: 124043:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9497.990741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9498.016746] Lustre: Skipped 1 previous similar message [ 9498.362934] Lustre: lustre-MDT0001: new disk, initializing [ 9498.660526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9498.763373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9498.817543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9508.196589] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9524.612672] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9533.540266] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9548.331541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9549.003164] Lustre: lustre-OST0000: new disk, initializing [ 9549.020114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9549.262815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9558.298804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9558.345374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9558.637497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9561.830245] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9577.409048] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9586.279414] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9586.905987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9595.883876] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9596.371967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9604.586166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9604.594629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9604.621720] LustreError: Skipped 1 previous similar message [ 9604.627938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9604.654490] Lustre: Skipped 1 previous similar message [ 9604.676666] Lustre: Skipped 1 previous similar message [ 9607.131213] Lustre: server umount lustre-OST0000 complete [ 9615.249170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9615.258103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9615.283067] Lustre: Skipped 1 previous similar message [ 9615.324290] Lustre: Skipped 1 previous similar message [ 9620.453434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9620.471524] Lustre: Skipped 1 previous similar message [ 9622.007231] Lustre: server umount lustre-MDT0000 complete [ 9629.666534] LustreError: 124055:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9629.737860] LustreError: 124055:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 9629.796216] LustreError: 124034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777003939 with bad export cookie 15115156298783558519 [ 9629.806724] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9629.821658] LustreError: 124034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9630.758468] Lustre: server umount lustre-MDT0001 complete [ 9647.098967] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 00:12:34 (1777003954) [ 9660.882323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9662.245866] LustreError: 128066:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9662.407857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9667.555119] LustreError: 128067:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9670.341098] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9672.694681] LustreError: 128066:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9677.801697] LustreError: 128067:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9687.873823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9688.037224] LustreError: 128066:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9688.093861] LustreError: 128066:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 9688.974032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9698.107585] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9708.910817] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9716.248912] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9723.875491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9723.886152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9723.915051] Lustre: Skipped 1 previous similar message [ 9726.820255] Lustre: server umount lustre-MDT0000 complete [ 9728.997757] LustreError: 128061:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9729.052023] LustreError: 128061:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 9734.013105] LustreError: 128047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777004043 with bad export cookie 15115156298783559401 [ 9734.036442] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9734.047233] LustreError: 128047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9734.704348] Lustre: server umount lustre-MDT0001 complete [ 9752.295447] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 00:14:19 (1777004059) [ 9766.673509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9782.624414] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9787.808143] LustreError: 130264:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.150@tcp: failed processing log, type 4: rc = -110 [ 9813.663943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9825.500761] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9836.858940] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9841.399407] Lustre: server umount lustre-OST0000 complete [ 9858.097637] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 00:16:05 (1777004165) [ 9870.048377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9871.401899] LustreError: 131599:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9871.581409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9879.185086] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9897.893703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9906.875678] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9918.334396] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9925.941041] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9939.041827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9939.982904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9940.018087] Lustre: Skipped 1 previous similar message [ 9953.275011] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9965.977885] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9983.792687] Lustre: server umount lustre-MDT0000 complete [ 9985.003182] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9985.012151] LustreError: 131599:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9985.056647] Lustre: Skipped 2 previous similar messages [ 9985.096604] LustreError: 131599:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 9992.063441] LustreError: 131581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777004301 with bad export cookie 15115156298783560066 [ 9992.103940] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9992.107811] LustreError: 131581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9993.036348] Lustre: server umount lustre-MDT0001 complete [10010.736761] Lustre: server umount lustre-OST0000 complete [10027.910812] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 00:18:55 (1777004335) [10041.570890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10042.711026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10051.137729] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10071.364412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10081.789290] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10093.112986] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10100.784947] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10114.297682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10125.581235] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10135.590809] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10137.672287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10142.964533] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10143.680292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10152.908582] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10153.312561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10158.050240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10158.084793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10158.104601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10158.160586] Lustre: Skipped 1 previous similar message [10158.205637] Lustre: Skipped 2 previous similar messages [10163.658858] Lustre: server umount lustre-OST0000 complete [10168.288849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10168.291360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10168.308687] Lustre: Skipped 1 previous similar message [10168.318199] Lustre: Skipped 2 previous similar messages [10174.275234] Lustre: server umount lustre-MDT0000 complete [10177.824799] LustreError: 134925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777004487 with bad export cookie 15115156298783561137 [10177.833448] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10177.840915] LustreError: 134925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10178.114425] Lustre: server umount lustre-MDT0001 complete [10188.704667] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 00:21:37 (1777004497) [10194.965763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10210.591770] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10215.715147] LustreError: 138740:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.150@tcp: failed processing log, type 4: rc = -110 [10241.444215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10241.466322] Lustre: Skipped 2 previous similar messages [10247.453874] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10252.816604] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10258.480784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10259.081339] LustreError: 138765:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10259.121906] LustreError: 138765:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 9 previous similar messages [10259.294618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [10263.101822] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10271.162654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10274.925670] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10279.243500] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10282.008881] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10285.062688] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10285.255425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10288.654221] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10288.933592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10291.171008] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10291.178397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10291.200663] Lustre: Skipped 1 previous similar message [10291.224084] Lustre: Skipped 5 previous similar messages [10296.825320] Lustre: server umount lustre-MDT0000 complete [10300.195470] LustreError: 138747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777004609 with bad export cookie 15115156298783562159 [10300.200828] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10300.212363] LustreError: 138747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10300.526360] Lustre: server umount lustre-MDT0001 complete [10314.103887] Lustre: server umount lustre-OST0000 complete [10321.974624] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 00:23:50 (1777004630) [10328.005062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10343.711758] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10348.832656] LustreError: 142563:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.150@tcp: failed processing log, type 4: rc = -110 [10380.221465] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10385.385974] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10391.818113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10392.581785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [10396.224963] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10405.655819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10409.852824] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10415.554445] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10419.503071] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10425.702512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10425.857195] Lustre: lustre-OST0001: new disk, initializing [10425.862979] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10431.141568] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10431.534461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10431.551946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10431.660937] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10438.775143] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10442.620557] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [10442.911947] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10446.945594] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10447.329136] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10452.474895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10452.482990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10452.508290] Lustre: Skipped 2 previous similar messages [10452.519415] Lustre: Skipped 3 previous similar messages [10452.530876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10455.297199] Lustre: server umount lustre-OST0000 complete [10462.697886] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10462.710584] LustreError: Skipped 1 previous similar message [10465.404113] Lustre: server umount lustre-OST0001 complete [10472.931775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10475.687446] Lustre: server umount lustre-MDT0000 complete [10479.472804] LustreError: 142571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777004789 with bad export cookie 15115156298783563111 [10479.479481] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10479.502703] LustreError: 142571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10479.874859] Lustre: server umount lustre-MDT0001 complete [10505.131573] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 00:26:53 (1777004813) [10506.873517] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [10508.969474] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [10511.301909] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 00:26:59 (1777004819) [10518.596821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10519.072150] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10519.116788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10519.363911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10519.376597] Lustre: Skipped 6 previous similar messages [10523.786984] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10532.559684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10532.789888] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10532.842369] Lustre: 148510:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10537.784106] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10543.936968] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10548.183350] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10553.901903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10554.126827] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10560.090818] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10560.658706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [10566.677828] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10570.616215] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10570.857432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10574.684825] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10574.865844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10580.961681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10580.977237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10580.996636] Lustre: Skipped 4 previous similar messages [10581.005684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10581.014692] Lustre: Skipped 4 previous similar messages [10582.772008] Lustre: server umount lustre-OST0000 complete [10604.964655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10606.515785] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10606.538717] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [10611.195976] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10615.881781] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10620.433668] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10621.934254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10621.959908] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [10622.756768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [10626.933602] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10627.203670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10630.518538] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff967b073b1000.ost_server_uuid 50 [10631.762610] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff967b073b1000.ost_server_uuid in FULL state after 0 sec [10632.034902] LustreError: 148517:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.50@tcp) returned error from blocking AST (req@00000000054ef194 x1863323428871040 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b57d842f/0xd1c3d291f980a6c1 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.50@tcp remote: 0x2e2b70ae0b0345d9 expref: 5 pid: 148518 timeout: 10732 lvb_type: 0 lru_score: 0 lru_type: 0 [10632.102118] LustreError: lustre-MDT0000: A client on nid 192.168.201.50@tcp was evicted due to a lock blocking callback time out: rc -107 [10632.123437] LustreError: 148507:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.50@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b57d842f/0xd1c3d291f980a6c1 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.50@tcp remote: 0x2e2b70ae0b0345d9 expref: 6 pid: 148518 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [10637.285781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10637.304604] LustreError: Skipped 2 previous similar messages [10641.259148] Lustre: server umount lustre-OST0000 complete [10650.941837] Lustre: server umount lustre-MDT0000 complete [10652.646078] LustreError: 148523:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10652.697206] LustreError: 148523:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 12 previous similar messages [10654.710842] LustreError: 149235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777004964 with bad export cookie 15115156298783564294 [10654.720187] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10654.743426] LustreError: 149235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10661.166227] Lustre: server umount lustre-MDT0001 complete [10671.390643] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10674.251067] Key type lgssc unregistered [10674.512998] LNet: 154006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10674.530261] LNetError: 154006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10674.547888] LNet: Removed LNI 192.168.201.150@tcp [10675.273676] Key type .llcrypt unregistered [10675.279103] Key type ._llcrypt unregistered [10694.420820] Key type ._llcrypt registered [10694.427668] Key type .llcrypt registered [10694.548727] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [10695.617995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10695.650649] alg: No test for adler32 (adler32-zlib) [10696.679574] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [10696.918655] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [10698.607242] Key type lgssc registered [10699.573078] Lustre: Echo OBD driver; http://www.lustre.org/ [10706.770627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10706.796922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10708.413948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10711.524787] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10718.525753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10718.690580] LustreError: 155382:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10718.719154] LustreError: 155382:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [10718.963522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10722.882466] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10727.919776] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10731.409605] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10737.597913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10738.095106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10744.056196] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10749.871136] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10753.459161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10760.295606] Lustre: Failing over lustre-MDT0000 [10760.722321] Lustre: server umount lustre-MDT0000 complete [10761.786216] LustreError: 155378:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10763.745495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10763.761747] LustreError: 155376:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10763.763924] Lustre: Skipped 2 previous similar messages [10763.800693] LustreError: 155376:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [10768.871319] LustreError: 156412:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10768.919937] LustreError: 156412:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [10772.726637] LustreError: 155362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.50@tcp arrived at 1777005082 with bad export cookie 3187689182107423331 [10773.987547] LustreError: 155406:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10774.023871] LustreError: 155406:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [10780.118545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10780.135018] Lustre: 154695:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777005073/real 1777005073] req@00000000ac76e97b x1863324801196928/t0(0) o400->MGC192.168.201.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777005089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10780.199970] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10785.252761] LustreError: 155382:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10785.276500] LustreError: 155382:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [10789.347870] LustreError: 154692:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000012411676 x1863324801202304/t0(0) o250->MGC192.168.201.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10789.799012] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10789.904451] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10794.305756] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10794.984331] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10794.991668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10796.769687] Lustre: lustre-MDT0000: Denying connection for new client eddd8761-c9f5-4278-becf-07f74e62679f (at 192.168.201.50@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [10805.731203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10805.743265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10805.763080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10810.352325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10810.357589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10811.680806] Lustre: server umount lustre-OST0000 complete [10815.947207] LustreError: 158757:0:(ldlm_lib.c:2986:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [10815.969359] Lustre: 158123:0:(ldlm_lib.c:2389:target_recovery_overseer()) recovery is aborted, evict exports in recovery [10815.990469] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [10816.021259] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [10816.043148] LustreError: 158123:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ad7167a1 x1863324801218048/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10816.091339] LustreError: 158123:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [10816.111875] LustreError: 158123:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [10816.134880] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [10816.455205] Lustre: server umount lustre-MDT0000 complete [10820.481303] LustreError: 155362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777005130 with bad export cookie 3187689182107423660 [10820.490728] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10820.492852] LustreError: 155382:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10820.492865] LustreError: 155382:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [10820.508695] LustreError: 155362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10822.862927] Lustre: server umount lustre-MDT0001 complete [10833.621971] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10836.842925] Key type lgssc unregistered [10837.270192] LNet: 159493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10837.303280] LNetError: 159493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10837.324862] LNet: Removed LNI 192.168.201.150@tcp [10838.453148] Key type .llcrypt unregistered [10838.464647] Key type ._llcrypt unregistered [10863.845853] Key type ._llcrypt registered [10863.854031] Key type .llcrypt registered [10864.009306] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [10865.299091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10865.389824] alg: No test for adler32 (adler32-zlib) [10866.595109] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [10866.917694] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [10868.639892] Key type lgssc registered [10870.152827] Lustre: Echo OBD driver; http://www.lustre.org/ [10879.517721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10879.561309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10881.268857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10885.294144] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10890.725602] LustreError: 160888:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10890.755684] LustreError: 160888:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [10893.908138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10894.696994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10899.632227] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10905.251101] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10908.578484] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10914.963638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10915.349383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10920.336532] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10925.450813] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10925.616745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10930.664457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10930.664591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10930.666246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10930.710413] Lustre: Skipped 1 previous similar message [10932.454923] Lustre: server umount lustre-OST0000 complete [10935.780168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10935.780573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10935.793188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10935.809866] Lustre: Skipped 1 previous similar message [10935.818298] Lustre: Skipped 1 previous similar message [10940.897103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10940.906056] Lustre: Skipped 2 previous similar messages [10946.017248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10946.026225] Lustre: Skipped 1 previous similar message [10950.111212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10950.296480] Lustre: server umount lustre-MDT0000 complete [10951.139569] LustreError: 160887:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10951.159052] LustreError: 160887:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [10953.057779] LustreError: 160869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777005262 with bad export cookie 5516630197144970746 [10953.062049] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10953.078049] LustreError: 160869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10953.411372] Lustre: server umount lustre-MDT0001 complete [10961.830729] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10963.981848] Key type lgssc unregistered [10964.262080] LNet: 164069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10964.272814] LNetError: 164069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10964.286098] LNet: Removed LNI 192.168.201.150@tcp [10964.967947] Key type .llcrypt unregistered [10964.973112] Key type ._llcrypt unregistered [10990.382990] Key type ._llcrypt registered [10990.387408] Key type .llcrypt registered [10990.500899] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [10991.423693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10991.522827] alg: No test for adler32 (adler32-zlib) [10992.524613] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [10992.720478] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [10994.367371] Key type lgssc registered [10995.389273] Lustre: Echo OBD driver; http://www.lustre.org/ [11002.442805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11002.467247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11004.013725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11007.730514] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11013.603781] LustreError: 165733:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11013.633060] LustreError: 165733:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11015.192409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11015.690295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11019.025485] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11023.981717] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11028.474120] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11034.857985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11035.181434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11040.374910] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11045.611429] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11048.494701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [11055.584861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11055.601196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11055.627363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11058.662772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11058.669409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11058.696855] Lustre: Skipped 1 previous similar message [11061.570357] Lustre: server umount lustre-OST0000 complete [11066.849885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11066.869439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11066.891197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11068.897905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11068.899741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11071.597804] Lustre: server umount lustre-MDT0000 complete [11074.017226] LustreError: 165733:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11074.037867] LustreError: 165733:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11075.072485] LustreError: 165712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777005384 with bad export cookie 10626096461934173467 [11075.081274] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11075.088281] LustreError: 165712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11075.398052] Lustre: server umount lustre-MDT0001 complete [11085.034453] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [11087.674773] Key type lgssc unregistered [11087.918759] LNet: 168909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11087.930302] LNetError: 168909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11087.946927] LNet: Removed LNI 192.168.201.150@tcp [11088.640404] Key type .llcrypt unregistered [11088.648424] Key type ._llcrypt unregistered [11109.958657] Key type ._llcrypt registered [11109.965776] Key type .llcrypt registered [11110.070981] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [11111.034361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11111.181360] alg: No test for adler32 (adler32-zlib) [11112.250308] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [11112.472132] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [11114.159209] Key type lgssc registered [11115.292357] Lustre: Echo OBD driver; http://www.lustre.org/ [11124.887063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11124.918282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11126.493785] Lustre: *** cfs_fail_loc=135, val=0*** [11126.593981] LustreError: 170415:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [11126.611829] LustreError: 170415:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.150@tcp: cfg command failed: rc = -2 [11126.633825] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [11126.633825] [11126.653001] LustreError: MGC192.168.201.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [11126.689802] LustreError: 170376:0:(tgt_mount.c:1838:server_start_targets()) failed to start server lustre-MDT0000: -2 [11126.706516] LustreError: 170376:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -2 [11126.730880] LustreError: 170376:0:(tgt_mount.c:2082:server_put_super()) no obd lustre-MDT0000 [11126.747124] LustreError: 170376:0:(tgt_mount.c:114:server_deregister_mount()) lustre-MDT0000 not registered [11126.892912] Lustre: server umount lustre-MDT0000 complete [11126.904666] LustreError: 170376:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [11132.525518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11133.038406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11136.839016] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11142.631419] LustreError: 170787:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11142.659953] LustreError: 170787:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11145.288052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11145.723768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11149.015807] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11153.535430] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11157.035810] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11161.061390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11161.078874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11161.109956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11163.106066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11163.107496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11168.226576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11168.251503] Lustre: Skipped 2 previous similar messages [11170.275379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11170.286505] Lustre: Skipped 1 previous similar message [11173.343190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11173.485630] Lustre: server umount lustre-MDT0000 complete [11175.402966] LustreError: 171849:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.432391] LustreError: 171849:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11176.882265] LustreError: 170768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777005486 with bad export cookie 6072912512818350397 [11176.884975] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11176.912406] LustreError: 170768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11177.257990] Lustre: server umount lustre-MDT0001 complete [11186.622462] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [11188.837814] Key type lgssc unregistered [11189.069889] LNet: 172867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11189.082908] LNetError: 172867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11189.105848] LNet: Removed LNI 192.168.201.150@tcp [11189.754649] Key type .llcrypt unregistered [11189.762493] Key type ._llcrypt unregistered [11207.880798] Key type ._llcrypt registered [11207.887776] Key type .llcrypt registered [11208.005908] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [11208.277331] Key type .llcrypt unregistered [11208.281840] Key type ._llcrypt unregistered [11224.051411] Key type ._llcrypt registered [11224.057451] Key type .llcrypt registered [11224.285787] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [11225.681710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11225.737293] alg: No test for adler32 (adler32-zlib) [11226.810557] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [11227.076078] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [11228.767830] Key type lgssc registered [11229.859037] Lustre: Echo OBD driver; http://www.lustre.org/ [11237.507544] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11237.555778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11253.215476] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11258.335486] LustreError: 174976:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.150@tcp: failed processing log, type 4: rc = -110 [11285.023851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11290.834605] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11295.782002] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11302.263818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11302.889160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11302.983513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [11306.648442] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11308.002580] LustreError: 175002:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11308.023633] LustreError: 175002:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11313.125366] LustreError: 175001:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11314.336614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11314.870619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11318.576534] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11322.839502] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11326.048712] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11328.700641] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [11333.602262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11333.614178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11333.638093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11335.140117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11335.156970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11335.186801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11339.191394] Lustre: server umount lustre-OST0000 complete [11345.377287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11345.391993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11345.415280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11345.425414] Lustre: Skipped 1 previous similar message [11349.213648] Lustre: server umount lustre-MDT0000 complete [11351.013314] LustreError: 175907:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11351.049246] LustreError: 175907:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11352.804420] LustreError: 177303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777005662 with bad export cookie 3574787523204489455 [11352.810176] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11352.830018] LustreError: 177303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11353.201898] Lustre: server umount lustre-MDT0001 complete [11364.389235] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [11366.930012] Key type lgssc unregistered [11367.219908] LNet: 178443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11367.231798] LNetError: 178443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11367.254938] LNet: Removed LNI 192.168.201.150@tcp [11368.010486] Key type .llcrypt unregistered [11368.018142] Key type ._llcrypt unregistered [11389.954841] Key type ._llcrypt registered [11389.961836] Key type .llcrypt registered [11390.069246] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [11391.122717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11391.152403] alg: No test for adler32 (adler32-zlib) [11392.212353] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [11392.477830] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [11394.143566] Key type lgssc registered [11395.205609] Lustre: Echo OBD driver; http://www.lustre.org/ [11401.965384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11402.004492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11403.515348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11406.694109] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11413.770035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11413.986605] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11414.134343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11417.331853] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11422.130386] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11425.236598] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11430.924340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11431.338484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11436.222628] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11441.269758] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11448.750010] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [11454.878634] Lustre: Failing over lustre-MDT0000 [11454.945275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11454.962432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11454.983370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11455.113569] Lustre: server umount lustre-MDT0000 complete [11457.124285] LustreError: 181803:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11457.164410] LustreError: 181803:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11459.044875] LustreError: 180545:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.045168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11459.078027] LustreError: 180545:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11459.092797] Lustre: Skipped 1 previous similar message [11462.183219] LustreError: 181803:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11467.301036] LustreError: 179834:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11467.336376] LustreError: 179834:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [11472.421083] LustreError: 180904:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11472.461422] LustreError: 180904:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [11473.230007] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11473.527986] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11473.902079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11474.013376] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11476.581489] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11478.177909] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11479.032254] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11479.089912] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11479.152964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [11482.951413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [11484.916454] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [11494.883687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11494.884805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11494.901811] LustreError: Skipped 1 previous similar message [11494.934472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11494.952028] Lustre: Skipped 1 previous similar message [11496.698206] Lustre: server umount lustre-OST0000 complete [11504.609950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11504.611108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11504.633497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11504.659120] Lustre: Skipped 2 previous similar messages [11506.855017] Lustre: server umount lustre-MDT0000 complete [11509.731274] LustreError: 180545:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11509.761400] LustreError: 180545:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11510.678018] LustreError: 179819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777005820 with bad export cookie 6159296366957275988 [11510.691552] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11510.708469] LustreError: 179819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11511.118798] Lustre: server umount lustre-MDT0001 complete [11521.028063] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [11523.447886] Key type lgssc unregistered [11523.705794] LNet: 184224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11523.717349] LNetError: 184224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11523.736348] LNet: Removed LNI 192.168.201.150@tcp [11524.826169] Key type .llcrypt unregistered [11524.831317] Key type ._llcrypt unregistered [11545.643746] Key type ._llcrypt registered [11545.650029] Key type .llcrypt registered [11545.751639] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [11546.897431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11546.967979] alg: No test for adler32 (adler32-zlib) [11548.090236] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [11548.403564] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [11550.095397] Key type lgssc registered [11551.411014] Lustre: Echo OBD driver; http://www.lustre.org/ [11558.267010] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11558.289870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11559.793700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11563.325821] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11570.146847] LustreError: 185615:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11570.175489] LustreError: 185615:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11570.591696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11571.060406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11575.287530] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11580.394725] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11583.529561] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11589.353051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11589.828505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11594.818808] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11599.430938] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11605.241872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [11611.615907] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [11616.730306] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [11626.582991] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [11637.216168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11637.222643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11637.232034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11641.319667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11641.320431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11641.321362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11642.432536] Lustre: server umount lustre-OST0000 complete [11646.436977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11646.438844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11646.456067] Lustre: Skipped 1 previous similar message [11646.474119] Lustre: Skipped 2 previous similar messages [11651.552814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11651.559341] Lustre: Skipped 1 previous similar message [11656.672881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11656.678828] Lustre: Skipped 1 previous similar message [11659.231178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11659.387313] Lustre: server umount lustre-MDT0000 complete [11661.793300] LustreError: 185614:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11661.814199] LustreError: 185614:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11662.110098] LustreError: 187834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777005971 with bad export cookie 3359971843916099641 [11662.116047] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11662.122159] LustreError: 187834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11662.348305] Lustre: server umount lustre-MDT0001 complete [11669.845459] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [11672.181721] Key type lgssc unregistered [11672.409056] LNet: 188982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11672.419239] LNetError: 188982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11672.434817] LNet: Removed LNI 192.168.201.150@tcp [11673.081419] Key type .llcrypt unregistered [11673.089221] Key type ._llcrypt unregistered [11693.712748] Key type ._llcrypt registered [11693.718206] Key type .llcrypt registered [11693.815388] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [11694.770502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11694.950117] alg: No test for adler32 (adler32-zlib) [11695.966945] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [11696.152124] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [11697.799229] Key type lgssc registered [11698.614541] Lustre: Echo OBD driver; http://www.lustre.org/ [11704.614415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11704.653119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11706.189789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11709.733926] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11715.554263] LustreError: 190375:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11715.579536] LustreError: 190375:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11716.078581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11716.487936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11719.618866] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11723.914568] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11727.091255] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11733.092991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11733.422391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11737.963417] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11742.839246] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11750.833761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [11757.791479] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [11768.772474] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [11777.318032] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [11785.442462] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [11796.960311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11796.970934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11796.993500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11797.987401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11798.002389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11798.025410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11800.044641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11800.059071] Lustre: Skipped 1 previous similar message [11801.815446] Lustre: server umount lustre-OST0000 complete [11805.159214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11805.191334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11805.200907] Lustre: Skipped 1 previous similar message [11810.273969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11810.285729] Lustre: Skipped 1 previous similar message [11810.991074] Lustre: server umount lustre-MDT0000 complete [11814.222813] LustreError: 192738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777006123 with bad export cookie 13708317584961316305 [11814.230698] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11814.244097] LustreError: 192738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11814.619542] Lustre: server umount lustre-MDT0001 complete [11824.136638] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [11826.801460] Key type lgssc unregistered [11827.108900] LNet: 195205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11827.124879] LNetError: 195205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11827.145158] LNet: Removed LNI 192.168.201.150@tcp [11827.956635] Key type .llcrypt unregistered [11827.960159] Key type ._llcrypt unregistered [11847.523949] Key type ._llcrypt registered [11847.530094] Key type .llcrypt registered [11847.655552] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [11848.426525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11848.585298] alg: No test for adler32 (adler32-zlib) [11849.603254] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [11849.850347] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [11851.551532] Key type lgssc registered [11852.735386] Lustre: Echo OBD driver; http://www.lustre.org/ [11861.170089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11861.209876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11862.864587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11866.264703] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11873.180302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11873.251967] LustreError: 196579:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11873.279257] LustreError: 196579:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11873.594136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11876.867366] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11881.461683] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11885.070909] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11890.360158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11890.699412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11894.906115] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11899.459911] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11908.144451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [11919.329104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11919.336975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11919.353907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11923.430609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11923.431958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11923.432785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11924.873002] Lustre: server umount lustre-OST0000 complete [11928.549016] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11928.550986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11928.575132] Lustre: Skipped 1 previous similar message [11928.588038] Lustre: Skipped 1 previous similar message [11933.664790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11933.671655] Lustre: Skipped 2 previous similar messages [11938.785318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11938.792091] Lustre: Skipped 1 previous similar message [11942.367358] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11942.502227] Lustre: server umount lustre-MDT0000 complete [11943.906097] LustreError: 196576:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11943.931563] LustreError: 196576:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11945.540455] LustreError: 196562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777006255 with bad export cookie 2379349507939719291 [11945.542488] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11945.558854] LustreError: 196562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11945.851851] Lustre: server umount lustre-MDT0001 complete [11955.304618] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [11957.391809] Key type lgssc unregistered [11957.616925] LNet: 199931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11957.625856] LNetError: 199931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11957.641425] LNet: Removed LNI 192.168.201.150@tcp [11958.206064] Key type .llcrypt unregistered [11958.208907] Key type ._llcrypt unregistered [11973.375554] Key type ._llcrypt registered [11973.380735] Key type .llcrypt registered [11973.473526] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [11974.566056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11974.590832] alg: No test for adler32 (adler32-zlib) [11975.616088] Lustre: Lustre: Build Version: 2.17.52_54_gb645d1c [11975.881513] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [11977.639470] Key type lgssc registered [11978.685519] Lustre: Echo OBD driver; http://www.lustre.org/ [11985.496763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11985.531817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11987.189264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11990.660662] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [11996.647392] LustreError: 201158:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11996.690953] LustreError: 201158:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [11998.300153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11998.808965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12002.094494] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [12006.650420] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12010.354965] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12016.336194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12016.792607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12021.742269] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [12026.089409] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12032.178905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [12047.844674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12047.847649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12047.847863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12047.865294] Lustre: Skipped 1 previous similar message [12051.915699] Lustre: server umount lustre-OST0000 complete [12055.011707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12055.020373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12055.037848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12055.049355] Lustre: Skipped 1 previous similar message [12058.082878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12058.083971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12063.205117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12063.220276] Lustre: Skipped 2 previous similar messages [12067.300940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12067.313956] Lustre: Skipped 1 previous similar message [12069.343210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12069.467099] Lustre: server umount lustre-MDT0000 complete [12072.417156] LustreError: 201159:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12072.444493] LustreError: 201159:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [12072.492661] LustreError: 201876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777006382 with bad export cookie 8448446946918521800 [12072.498610] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12072.512112] LustreError: 201876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12072.750331] Lustre: server umount lustre-MDT0001 complete [12081.870488] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [12084.224168] Key type lgssc unregistered [12084.471051] LNet: 204509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12084.482430] LNetError: 204509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12084.501393] LNet: Removed LNI 192.168.201.150@tcp [12085.316241] Key type .llcrypt unregistered [12085.329802] Key type ._llcrypt unregistered