[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508602632 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002373] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.007289] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010137] LSM: Security Framework initializing [ 0.011045] Yama: becoming mindful. [ 0.012029] SELinux: Initializing. [ 0.013084] *** VALIDATE selinux *** [ 0.021825] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026556] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027012] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028144] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029130] *** VALIDATE tmpfs *** [ 0.030491] *** VALIDATE proc *** [ 0.032061] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.035106] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042162] debug: unmapping init [mem 0xffffffffb4259000-0xffffffffb4260fff] [ 0.045157] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046745] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047030] ... version: 2 [ 0.048018] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054282] rcu: Hierarchical SRCU implementation. [ 0.056197] smp: Bringing up secondary CPUs ... [ 0.057543] x86: Booting SMP configuration: [ 0.058032] .... node #0, CPUs: #1 #2 #3 [ 0.061154] smp: Brought up 1 node, 4 CPUs [ 0.062907] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131781] node 0 deferred pages initialised in 66ms [ 0.135143] devtmpfs: initialized [ 0.136288] x86/mm: Memory block size: 128MB [ 0.138753] gcov: version magic: 0x41383552 [ 0.140305] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142233] pinctrl core: initialized pinctrl subsystem [ 0.143203] [ 0.143843] ************************************************************* [ 0.144016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145016] ** ** [ 0.146012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147014] ** ** [ 0.148015] ** This means that this kernel is built to expose internal ** [ 0.149014] ** IOMMU data structures, which may compromise security on ** [ 0.150018] ** your system. ** [ 0.151013] ** ** [ 0.152014] ** If you see this message and you are not debugging the ** [ 0.153013] ** kernel, report this immediately to your vendor! ** [ 0.154014] ** ** [ 0.155013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156014] ************************************************************* [ 0.157706] NET: Registered protocol family 16 [ 0.158443] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.159067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.160063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.161460] cpuidle: using governor menu [ 0.163718] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.166373] PCI: Using configuration type 1 for base access [ 0.169160] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.177105] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.178034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.180092] cryptd: max_cpu_qlen set to 1000 [ 0.183250] ACPI: Added _OSI(Module Device) [ 0.184012] ACPI: Added _OSI(Processor Device) [ 0.185017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.186015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.189978] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.191519] ACPI: Interpreter enabled [ 0.193051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.194010] ACPI: Using IOAPIC for interrupt routing [ 0.195120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.196362] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.204998] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.206027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.207017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.208082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.210347] acpiphp: Slot [2] registered [ 0.211095] acpiphp: Slot [5] registered [ 0.212121] acpiphp: Slot [6] registered [ 0.213109] acpiphp: Slot [7] registered [ 0.214121] acpiphp: Slot [8] registered [ 0.215113] acpiphp: Slot [9] registered [ 0.216235] acpiphp: Slot [10] registered [ 0.218191] acpiphp: Slot [3] registered [ 0.219090] acpiphp: Slot [4] registered [ 0.221117] acpiphp: Slot [11] registered [ 0.223163] acpiphp: Slot [12] registered [ 0.224136] acpiphp: Slot [13] registered [ 0.226117] acpiphp: Slot [14] registered [ 0.227159] acpiphp: Slot [15] registered [ 0.229143] acpiphp: Slot [16] registered [ 0.231194] acpiphp: Slot [17] registered [ 0.233121] acpiphp: Slot [18] registered [ 0.234207] acpiphp: Slot [19] registered [ 0.236124] acpiphp: Slot [20] registered [ 0.238148] acpiphp: Slot [21] registered [ 0.240121] acpiphp: Slot [22] registered [ 0.242135] acpiphp: Slot [23] registered [ 0.243136] acpiphp: Slot [24] registered [ 0.245127] acpiphp: Slot [25] registered [ 0.247200] acpiphp: Slot [26] registered [ 0.248122] acpiphp: Slot [27] registered [ 0.250131] acpiphp: Slot [28] registered [ 0.252217] acpiphp: Slot [29] registered [ 0.253119] acpiphp: Slot [30] registered [ 0.255201] acpiphp: Slot [31] registered [ 0.257109] PCI host bridge to bus 0000:00 [ 0.258035] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.261025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.263027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.266030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.269057] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.272029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.274170] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.278012] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.282336] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.294649] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.298437] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.302018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.304022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.307033] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.310505] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.313752] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.316049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.319746] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.325016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.338017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.343017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.348893] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.363021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.376021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.403018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.413656] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.423019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.433019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.458019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.470460] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.478020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.489018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.515020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.530284] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.540018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.549018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.572020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.584473] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.591017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.599020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.616018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.630509] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.636013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.643015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.657028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.670222] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.672364] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.675367] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.677348] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.681270] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.686108] iommu: Default domain type: Passthrough [ 0.688325] SCSI subsystem initialized [ 0.690156] ACPI: bus type USB registered [ 0.692107] usbcore: registered new interface driver usbfs [ 0.694084] usbcore: registered new interface driver hub [ 0.696080] usbcore: registered new device driver usb [ 0.697172] pps_core: LinuxPPS API ver. 1 registered [ 0.699018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.702075] PTP clock support registered [ 0.704073] EDAC MC: Ver: 3.0.0 [ 0.706147] PCI: Using ACPI for IRQ routing [ 0.707908] NetLabel: Initializing [ 0.710013] NetLabel: domain hash size = 128 [ 0.711010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.713080] NetLabel: unlabeled traffic allowed by default [ 0.715145] vgaarb: loaded [ 0.717231] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.718034] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.727148] clocksource: Switched to clocksource kvm-clock [ 0.818450] VFS: Disk quotas dquot_6.6.0 [ 0.820696] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.822341] *** VALIDATE ramfs *** [ 0.823169] *** VALIDATE hugetlbfs *** [ 0.824146] pnp: PnP ACPI init [ 0.825619] pnp: PnP ACPI: found 6 devices [ 0.838711] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.840910] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.842241] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.843611] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.845095] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.846527] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.848304] NET: Registered protocol family 2 [ 0.849992] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.853274] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.857270] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.862349] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.864788] TCP: Hash tables configured (established 65536 bind 65536) [ 0.866777] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.868582] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.870419] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.872364] NET: Registered protocol family 1 [ 0.874346] RPC: Registered named UNIX socket transport module. [ 0.875623] RPC: Registered udp transport module. [ 0.876494] RPC: Registered tcp transport module. [ 0.877606] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.878986] NET: Registered protocol family 44 [ 0.880013] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.881242] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.882482] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.883959] PCI: CLS 0 bytes, default 64 [ 0.885771] Unpacking initramfs... [ 2.222937] debug: unmapping init [mem 0xffff9903bcc54000-0xffff9903bffbffff] [ 2.227096] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.229311] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.232254] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.699512] Initialise system trusted keyrings [ 2.701079] Key type blacklist registered [ 2.702390] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.710485] zbud: loaded [ 2.713568] *** VALIDATE nfs *** [ 2.714556] *** VALIDATE nfs4 *** [ 2.715583] pstore: using deflate compression [ 2.718193] Platform Keyring initialized [ 2.795757] NET: Registered protocol family 38 [ 2.797711] Key type asymmetric registered [ 2.800307] Asymmetric key parser 'x509' registered [ 2.802300] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.805485] io scheduler mq-deadline registered [ 2.808138] io scheduler kyber registered [ 2.809941] io scheduler bfq registered [ 2.812847] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.815651] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.818531] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.821569] ACPI: Power Button [PWRF] [ 2.826436] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.831927] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.844678] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.851118] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.868067] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.895507] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.925468] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.930585] Non-volatile memory driver v1.3 [ 2.932475] Linux agpgart interface v0.103 [ 2.965035] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.968168] vda: detected capacity change from 0 to 76791808 [ 2.984274] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.987751] vdb: detected capacity change from 0 to 1073741824 [ 3.003349] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.006173] vdc: detected capacity change from 0 to 2621440000 [ 3.021345] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.023357] vdd: detected capacity change from 0 to 2621440000 [ 3.038824] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.041109] vde: detected capacity change from 0 to 4294967296 [ 3.055130] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.058219] vdf: detected capacity change from 0 to 4294967296 [ 3.065046] libphy: Fixed MDIO Bus: probed [ 3.076171] usbcore: registered new interface driver usbserial_generic [ 3.078379] usbserial: USB Serial support registered for generic [ 3.080518] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.084454] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.086073] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.088947] mousedev: PS/2 mouse device common for all mice [ 3.091576] rtc_cmos 00:05: RTC can wake from S4 [ 3.094436] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.095975] rtc_cmos 00:05: registered as rtc0 [ 3.099478] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.102567] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.103410] intel_pstate: CPU model not supported [ 3.109679] hid: raw HID events driver (C) Jiri Kosina [ 3.112536] usbcore: registered new interface driver usbhid [ 3.114048] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.115330] usbhid: USB HID core driver [ 3.115492] drop_monitor: Initializing network drop monitor service [ 3.115620] Initializing XFRM netlink socket [ 3.115959] NET: Registered protocol family 10 [ 3.117232] Segment Routing with IPv6 [ 3.126323] NET: Registered protocol family 17 [ 3.128155] mpls_gso: MPLS GSO support [ 3.133524] RAS: Correctable Errors collector initialized. [ 3.135480] AVX version of gcm_enc/dec engaged. [ 3.136210] AES CTR mode by8 optimization enabled [ 3.188734] sched_clock: Marking stable (3188708641, 0)->(4145682964, -956974323) [ 3.191136] registered taskstats version 1 [ 3.192387] Loading compiled-in X.509 certificates [ 3.193661] zswap: loaded using pool lzo/zbud [ 3.211258] Key type big_key registered [ 3.220521] Key type encrypted registered [ 3.221865] ima: No TPM chip found, activating TPM-bypass! [ 3.223715] ima: Allocated hash algorithm: sha1 [ 3.225530] ima: No architecture policies found [ 3.227100] evm: Initialising EVM extended attributes: [ 3.228812] evm: security.selinux [ 3.229941] evm: security.ima [ 3.230921] evm: security.capability [ 3.232119] evm: HMAC attrs: 0x1 [ 3.234268] rtc_cmos 00:05: setting system clock to 2026-09-10 14:17:13 UTC (1789049833) [ 3.240359] debug: unmapping init [mem 0xffffffffb5203000-0xffffffffb53fffff] [ 3.243231] debug: unmapping init [mem 0xffffffffb3f82000-0xffffffffb4258fff] [ 3.252090] Write protecting the kernel read-only data: 28672k [ 3.255773] debug: unmapping init [mem 0xffffffffb2603000-0xffffffffb27fffff] [ 3.258576] debug: unmapping init [mem 0xffffffffb2f14000-0xffffffffb2ffffff] [ 3.289846] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.297299] systemd[1]: Detected virtualization kvm. [ 3.299205] systemd[1]: Detected architecture x86-64. [ 3.300318] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.324266] systemd[1]: No hostname configured. [ 3.325421] systemd[1]: Set hostname to . [ 3.327174] random: systemd: uninitialized urandom read (16 bytes read) [ 3.329364] systemd[1]: Initializing machine ID from random generator. [ 3.443268] random: systemd: uninitialized urandom read (16 bytes read) [ 3.445936] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.450258] random: systemd: uninitialized urandom read (16 bytes read) [ 3.452599] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.455420] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.042569] device-mapper: uevent: version 1.0.3 [ 4.044839] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.812368] virtio_net virtio0 ens2: renamed from eth0 [ 4.820721] random: fast init done [ 4.886886] scsi host0: ata_piix [ 4.889054] scsi host1: ata_piix [ 4.892925] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.895217] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.653266] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.741775] random: crng init done [ 9.743302] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.200947] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.416680] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.719110] SELinux: Disabled at runtime. [ 11.783089] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.793222] systemd[1]: Detected virtualization kvm. [ 11.795297] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.332243] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.335464] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.341851] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.347431] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.351895] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.362736] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.371064] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK [0[ 12.477600] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.905194] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.255982] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.286915] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.452910] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.487440] EDAC sbridge: Ver: 1.1.2 [ 15.291116] Key type dns_resolver registered [ 15.599334] NFS: Registering the id_resolver key type [ 15.601277] Key type id_resolver registered [ 15.602962] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg231-server login: [ 56.885925] libcfs: loading out-of-tree module taints kernel. [ 56.969070] Key type ._llcrypt registered [ 56.971858] Key type .llcrypt registered [ 57.129868] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 80.894744] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 83.022884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.035516] alg: No test for adler32 (adler32-zlib) [ 84.746402] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 86.274213] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 88.209702] Key type lgssc registered [ 90.509160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 110.720901] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.585128] hrtimer: interrupt took 5023434 ns [ 157.308338] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 173.613910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 173.721459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 175.258473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 175.351903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 175.572296] Lustre: lustre-MDT0000: new disk, initializing [ 175.781086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 175.824896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 182.463639] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.307776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.502975] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 200.566480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 200.570710] Lustre: Skipped 1 previous similar message [ 200.669853] Lustre: lustre-MDT0001: new disk, initializing [ 200.798948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.842492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 200.855363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 207.391437] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.431822] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 226.141581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.587612] Lustre: lustre-OST0000: new disk, initializing [ 226.593352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 226.604113] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.699223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.979782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 231.995293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 232.072150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 235.162552] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.330576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.462617] Lustre: lustre-OST0001: new disk, initializing [ 251.465632] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.472330] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.561205] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 259.075084] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.673409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 260.699751] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 260.762845] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 271.561125] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 278.910263] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 284.663621] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing check_logdir /tmp/testlogs/ [ 291.061525] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing yml_node [ 295.198273] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 297.704423] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 301.289299] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 303.447139] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 10:22:12 EDT 2026 [ 322.235797] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 324.215948] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 343.009328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.010622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 343.021450] Lustre: Skipped 3 previous similar messages [ 343.033164] Lustre: Skipped 3 previous similar messages [ 348.130079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 348.139767] Lustre: Skipped 3 previous similar messages [ 353.255336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.262448] Lustre: Skipped 3 previous similar messages [ 354.272102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 354.548023] Lustre: server umount lustre-MDT0000 complete [ 358.202552] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050188 with bad export cookie 1157947161591919234 [ 358.206286] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.212066] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 358.369335] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.381571] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 358.381904] Lustre: Skipped 1 previous similar message [ 363.499078] LustreError: 12155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.501111] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.522431] LustreError: 12155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 363.526705] Lustre: Skipped 2 previous similar messages [ 364.912837] Lustre: server umount lustre-MDT0001 complete [ 368.729922] Lustre: server umount lustre-OST0000 complete [ 372.311086] Lustre: server umount lustre-OST0001 complete [ 379.562718] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 389.990839] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 433.064624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.408714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 433.441382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 433.549383] Lustre: lustre-MDT0000: new disk, initializing [ 433.676051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 433.712923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 439.025653] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 451.383277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.552540] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 451.616988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 451.621358] Lustre: Skipped 1 previous similar message [ 451.741235] Lustre: lustre-MDT0001: new disk, initializing [ 451.946344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 451.997451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 452.010123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 457.690753] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 469.292070] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 476.149632] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 484.428083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 484.794963] Lustre: lustre-OST0000: new disk, initializing [ 484.800785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 484.812612] Lustre: 17050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 484.959748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 486.398216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 486.413589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 486.543388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 492.487561] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 502.745219] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 507.993945] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 508.213636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 513.474303] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 513.730138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 517.092942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 517.105696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.137794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.456713] Lustre: server umount lustre-OST0000 complete [ 527.841288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.852901] Lustre: Skipped 2 previous similar messages [ 532.913720] Lustre: server umount lustre-MDT0000 complete [ 532.971681] LustreError: 15090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.992457] LustreError: 15090:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 537.062213] LustreError: 15091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.075693] LustreError: 15091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 537.497773] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050367 with bad export cookie 1157947161591920767 [ 537.501949] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.506198] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 538.043116] Lustre: server umount lustre-MDT0001 complete [ 541.501957] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 10:26:10 (1789050370) [ 549.884335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.358051] LustreError: 19314:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.419993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.115628] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 555.491049] LustreError: 19315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.126113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.189626] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 577.208311] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 583.532325] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 591.248542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 598.442698] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 606.069287] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 620.513326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 620.521834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.545351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 620.547045] Lustre: Skipped 3 previous similar messages [ 623.585314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 625.912696] Lustre: server umount lustre-OST0000 complete [ 630.241331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 630.246996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.253543] Lustre: Skipped 1 previous similar message [ 636.249778] Lustre: server umount lustre-MDT0000 complete [ 638.946123] LustreError: 19314:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.973429] LustreError: 19314:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 639.704679] LustreError: 19295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050469 with bad export cookie 1157947161591921649 [ 639.711787] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.722284] LustreError: 19295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 640.259038] Lustre: server umount lustre-MDT0001 complete [ 652.556608] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 655.541553] Key type lgssc unregistered [ 655.854746] LNet: 22868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.869115] LNetError: 22868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.900888] LNet: Removed LNI 192.168.202.131@tcp [ 656.869230] Key type .llcrypt unregistered [ 656.870882] Key type ._llcrypt unregistered [ 681.907061] Key type ._llcrypt registered [ 681.910296] Key type .llcrypt registered [ 682.098501] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 683.726408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 683.885597] alg: No test for adler32 (adler32-zlib) [ 684.944677] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 685.284220] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 687.059158] Key type lgssc registered [ 688.151916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 696.669834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 696.716411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.544441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 703.735307] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 708.070954] LustreError: 24279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.108454] LustreError: 24279:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 711.884381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.193369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 716.927132] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 724.325749] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 730.816583] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 738.042642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 738.373862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 743.529867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 744.042787] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 751.741949] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 767.460994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 767.466685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.472991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 767.488214] Lustre: Skipped 1 previous similar message [ 772.582892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 772.590211] Lustre: Skipped 1 previous similar message [ 773.017963] Lustre: server umount lustre-OST0000 complete [ 777.706214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 777.708130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 777.715898] Lustre: Skipped 1 previous similar message [ 777.731986] Lustre: Skipped 1 previous similar message [ 782.806329] Lustre: server umount lustre-MDT0000 complete [ 782.820258] LustreError: 25283:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 782.847367] LustreError: 25283:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 787.080497] LustreError: 24260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050617 with bad export cookie 10499295671723247122 [ 787.093759] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 787.105890] LustreError: 24260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 787.464096] Lustre: server umount lustre-MDT0001 complete [ 798.852147] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 801.434503] Key type lgssc unregistered [ 801.742907] LNet: 28254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.751820] LNetError: 28254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.795755] LNet: Removed LNI 192.168.202.131@tcp [ 803.735241] Key type .llcrypt unregistered [ 803.737511] Key type ._llcrypt unregistered [ 827.888324] Key type ._llcrypt registered [ 827.891507] Key type .llcrypt registered [ 828.027733] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 829.181013] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 829.263864] alg: No test for adler32 (adler32-zlib) [ 830.623387] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 830.892442] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 832.600481] Key type lgssc registered [ 833.454377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 841.192949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 841.214096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 842.730512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.025690] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 851.944288] LustreError: 29665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 851.989130] LustreError: 29665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 857.057332] LustreError: 29666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 857.647043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.084567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 862.761489] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 870.373071] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 875.993464] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 887.016838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 887.286495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 888.309381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 893.064307] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 899.065152] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 911.863115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 911.869940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.891189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 913.893993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.910318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 916.185993] Lustre: server umount lustre-OST0000 complete [ 924.139225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.153392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 924.154878] Lustre: Skipped 1 previous similar message [ 924.167322] Lustre: Skipped 2 previous similar messages [ 929.249359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 929.263040] Lustre: Skipped 1 previous similar message [ 934.372276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 934.372785] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 934.388580] Lustre: Skipped 1 previous similar message [ 934.548804] Lustre: server umount lustre-MDT0000 complete [ 938.695449] LustreError: 29645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050768 with bad export cookie 14337963799760240442 [ 938.696380] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 938.712404] LustreError: 29645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 939.088856] Lustre: server umount lustre-MDT0001 complete [ 949.465996] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 952.346606] Key type lgssc unregistered [ 952.712943] LNet: 33544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.722142] LNetError: 33544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.740875] LNet: Removed LNI 192.168.202.131@tcp [ 953.504552] Key type .llcrypt unregistered [ 953.507699] Key type ._llcrypt unregistered [ 977.631137] Key type ._llcrypt registered [ 977.634167] Key type .llcrypt registered [ 977.742629] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 978.823263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 978.863179] alg: No test for adler32 (adler32-zlib) [ 979.907525] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 980.160790] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 981.840246] Key type lgssc registered [ 983.124554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 991.354940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 991.372729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.934285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.836220] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1003.489169] LustreError: 34953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1003.506796] LustreError: 34953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1005.621869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.129400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1011.072901] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1020.131964] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1026.743493] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1033.660529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1033.982804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1038.086182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1040.634168] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1048.518768] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1061.862359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.896939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1061.901527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1065.550661] Lustre: server umount lustre-OST0000 complete [ 1072.098300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.105331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.121128] Lustre: Skipped 2 previous similar messages [ 1072.132131] Lustre: Skipped 2 previous similar messages [ 1075.058934] Lustre: server umount lustre-MDT0000 complete [ 1077.217532] LustreError: 34953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.231927] LustreError: 34953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1079.012100] LustreError: 34934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050909 with bad export cookie 484121116156799298 [ 1079.020886] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1079.026459] LustreError: 34934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1079.310152] Lustre: server umount lustre-MDT0001 complete [ 1087.376771] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1089.993188] Key type lgssc unregistered [ 1090.275606] LNet: 38487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.281819] LNetError: 38487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.310119] LNet: Removed LNI 192.168.202.131@tcp [ 1091.121486] Key type .llcrypt unregistered [ 1091.123373] Key type ._llcrypt unregistered [ 1116.885242] Key type ._llcrypt registered [ 1116.888345] Key type .llcrypt registered [ 1117.017642] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1117.779563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1117.813196] alg: No test for adler32 (adler32-zlib) [ 1118.797242] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 1119.003583] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1120.773081] Key type lgssc registered [ 1121.796910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1129.488973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1129.500654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.172991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1135.227764] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1141.733402] LustreError: 39897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1141.760539] LustreError: 39897:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1143.980502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1144.312704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1148.843542] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1156.798187] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1162.798481] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1169.485717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1170.012958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.116897] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1177.218296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1183.488400] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1194.981239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1194.986721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1194.998655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1195.012622] Lustre: Skipped 1 previous similar message [ 1195.712561] Lustre: server umount lustre-OST0000 complete [ 1215.969404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1215.973175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.992365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.999922] Lustre: Skipped 1 previous similar message [ 1220.577796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1220.583283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.697442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.703495] Lustre: Skipped 2 previous similar messages [ 1230.305814] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1230.578687] Lustre: server umount lustre-MDT0000 complete [ 1230.817958] LustreError: 39896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.826274] LustreError: 39896:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1234.376553] LustreError: 39876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051064 with bad export cookie 5200490284890612335 [ 1234.385965] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.051528] Lustre: server umount lustre-MDT0001 complete [ 1251.074581] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1254.379548] Key type lgssc unregistered [ 1254.666317] LNet: 43604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1254.673655] LNetError: 43604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1254.694101] LNet: Removed LNI 192.168.202.131@tcp [ 1255.488697] Key type .llcrypt unregistered [ 1255.495563] Key type ._llcrypt unregistered [ 1278.016648] Key type ._llcrypt registered [ 1278.018222] Key type .llcrypt registered [ 1278.102638] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1279.275868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1279.325721] alg: No test for adler32 (adler32-zlib) [ 1280.476273] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 1280.878049] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1282.586049] Key type lgssc registered [ 1283.936516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1292.986752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1292.999591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.694331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1299.236834] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1304.034078] LustreError: 45010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.056268] LustreError: 45010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1307.293617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.525572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1311.525652] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1318.037857] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1323.998506] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1331.562517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1332.204154] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1332.503435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.873121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1343.226621] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1351.751983] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1358.075042] Lustre: server umount lustre-MDT0000 complete [ 1358.818631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1358.833380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.864770] LustreError: 45738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.344333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.369158] Lustre: Skipped 1 previous similar message [ 1362.264905] LustreError: 44989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051192 with bad export cookie 13870973128882632478 [ 1362.279726] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1364.003559] LustreError: 45005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.043828] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.31@tcp (stopping) [ 1364.044421] LustreError: 45005:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1364.452690] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.459698] LustreError: 46252:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.460702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1367.470517] LustreError: 46252:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1367.473805] Lustre: Skipped 1 previous similar message [ 1368.763707] Lustre: server umount lustre-MDT0001 complete [ 1391.491390] Lustre: server umount lustre-OST0000 complete [ 1411.431757] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1414.474196] Key type lgssc unregistered [ 1414.799058] LNet: 48869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1414.807725] LNetError: 48869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1414.837623] LNet: Removed LNI 192.168.202.131@tcp [ 1415.777470] Key type .llcrypt unregistered [ 1415.780152] Key type ._llcrypt unregistered [ 1443.503633] Key type ._llcrypt registered [ 1443.506994] Key type .llcrypt registered [ 1443.608893] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1444.863063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1444.877943] alg: No test for adler32 (adler32-zlib) [ 1445.946330] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 1446.278769] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1447.968430] Key type lgssc registered [ 1449.633263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1458.846591] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1458.887566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1474.592529] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1479.713499] LustreError: 50256:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 1506.336401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1513.699704] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1521.669926] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1687.224101] Lustre: server umount lustre-OST0000 complete [ 1706.243353] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1709.979734] Key type lgssc unregistered [ 1710.229703] LNet: 51929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.238817] LNetError: 51929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1711.271731] LNet: Removed LNI 192.168.202.131@tcp [ 1712.172577] Key type .llcrypt unregistered [ 1712.175921] Key type ._llcrypt unregistered [ 1735.014898] Key type ._llcrypt registered [ 1735.018098] Key type .llcrypt registered [ 1735.157845] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1736.589676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1736.624678] alg: No test for adler32 (adler32-zlib) [ 1737.869203] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 1738.154689] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1739.880264] Key type lgssc registered [ 1741.210889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1749.581466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1749.596571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.075357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1756.525910] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1761.260019] LustreError: 53343:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1761.300937] LustreError: 53343:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1766.111336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.512978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1770.951917] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1777.964604] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1783.772947] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1790.752522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1790.952907] LustreError: 55164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.099349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.242889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1798.526126] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1806.880618] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1811.937339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1811.948910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1811.966685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1811.978248] Lustre: Skipped 1 previous similar message [ 1816.332564] Lustre: server umount lustre-OST0000 complete [ 1822.178343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.184522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1822.197053] Lustre: Skipped 2 previous similar messages [ 1822.212016] Lustre: Skipped 1 previous similar message [ 1825.927357] Lustre: server umount lustre-MDT0000 complete [ 1827.298909] LustreError: 53339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1827.314504] LustreError: 53339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1830.203361] LustreError: 53323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051660 with bad export cookie 240037751147406871 [ 1830.210259] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.216632] LustreError: 53323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1831.282026] Lustre: server umount lustre-MDT0001 complete [ 1846.909368] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1849.745275] Key type lgssc unregistered [ 1850.173303] LNet: 56875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.180441] LNetError: 56875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1850.195718] LNet: Removed LNI 192.168.202.131@tcp [ 1851.231286] Key type .llcrypt unregistered [ 1851.234171] Key type ._llcrypt unregistered [ 1874.742285] Key type ._llcrypt registered [ 1874.743552] Key type .llcrypt registered [ 1874.909917] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1876.383325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1876.430956] alg: No test for adler32 (adler32-zlib) [ 1877.682763] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 1877.976713] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1879.736500] Key type lgssc registered [ 1880.726709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1887.355655] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1887.371130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1902.944395] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1908.064441] LustreError: 58258:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 1934.753327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.482761] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1946.027588] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1951.783344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.061544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1952.124973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1956.933782] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1957.368183] LustreError: 58283:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1957.378096] LustreError: 58283:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1962.465466] LustreError: 58284:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.667860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.949169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.573507] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1977.754361] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1983.935786] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1988.065534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1988.079506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.094995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1990.628739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.631926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1990.633685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1991.837723] Lustre: server umount lustre-OST0000 complete [ 2016.225864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2016.229929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.239732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.269175] Lustre: Skipped 1 previous similar message [ 2016.275250] Lustre: Skipped 1 previous similar message [ 2019.533225] Lustre: server umount lustre-MDT0000 complete [ 2021.349177] LustreError: 60998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.368242] LustreError: 60998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2023.368897] LustreError: 58264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051853 with bad export cookie 12456119892852140363 [ 2023.372297] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.378521] LustreError: 58264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2029.907867] Lustre: server umount lustre-MDT0001 complete [ 2042.834991] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2046.440887] Key type lgssc unregistered [ 2046.691799] LNet: 62005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.702452] LNetError: 62005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.713319] LNet: Removed LNI 192.168.202.131@tcp [ 2047.463322] Key type .llcrypt unregistered [ 2047.465241] Key type ._llcrypt unregistered [ 2073.140286] Key type ._llcrypt registered [ 2073.143300] Key type .llcrypt registered [ 2073.287469] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2074.605549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2074.633850] alg: No test for adler32 (adler32-zlib) [ 2075.614281] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2075.881839] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2077.577660] Key type lgssc registered [ 2078.918309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2087.358986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2087.384518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.761865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.275729] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2098.149684] LustreError: 63417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.175178] LustreError: 63417:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2102.332453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2102.638445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2106.947635] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2115.244871] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2120.443654] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2126.764624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2127.285798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.437862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2133.713159] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2139.989551] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2159.087635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.095572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2159.103545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2159.119390] Lustre: Skipped 1 previous similar message [ 2159.148301] LustreError: Skipped 1 previous similar message [ 2163.304658] Lustre: server umount lustre-OST0000 complete [ 2169.330683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.336909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2169.347390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.347399] Lustre: Skipped 1 previous similar message [ 2169.352181] Lustre: Skipped 1 previous similar message [ 2172.754689] Lustre: server umount lustre-MDT0000 complete [ 2174.443656] LustreError: 64658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.456136] LustreError: 64658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2176.217505] LustreError: 64151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052006 with bad export cookie 3560601150887766019 [ 2176.220942] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2176.227328] LustreError: 64151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2176.558428] Lustre: server umount lustre-MDT0001 complete [ 2188.699925] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2190.939208] Key type lgssc unregistered [ 2191.180227] LNet: 66968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.184431] LNetError: 66968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.194867] LNet: Removed LNI 192.168.202.131@tcp [ 2191.970952] Key type .llcrypt unregistered [ 2191.972647] Key type ._llcrypt unregistered [ 2223.320448] Key type ._llcrypt registered [ 2223.327914] Key type .llcrypt registered [ 2223.430324] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2224.726344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.920267] alg: No test for adler32 (adler32-zlib) [ 2226.159513] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2226.408498] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2228.072197] Key type lgssc registered [ 2229.255744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2237.785469] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 10:54:26 (1789052066) [ 2244.932784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2244.960775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2246.375938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.329572] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2255.860784] LustreError: 68667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.878916] LustreError: 68667:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2260.257702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.610799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2265.289457] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2272.829295] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2277.984973] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2284.173381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2284.410394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.288198] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2291.635622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2296.893327] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2304.226814] Lustre: Failing over lustre-MDT0000 [ 2304.615974] LustreError: 70582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.757314] Lustre: server umount lustre-MDT0000 complete [ 2306.529429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2306.540713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.561684] LustreError: 70100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.049645] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.077911] Lustre: Skipped 1 previous similar message [ 2309.683081] LustreError: 68662:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.696785] LustreError: 68662:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2314.785395] LustreError: 68663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.808105] LustreError: 68663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2315.134451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.290496] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.410690] Lustre: *** cfs_fail_loc=135, val=0*** [ 2315.531906] LustreError: 71754:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2315.536402] LustreError: 71754:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.131@tcp: cfg command failed: rc = -2 [ 2315.541029] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2315.541029] [ 2315.546190] LustreError: MGC192.168.202.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2315.559761] LustreError: 71741:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2315.571735] LustreError: 71741:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2315.582754] LustreError: 71741:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2315.588528] LustreError: 71741:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2315.629308] Lustre: server umount lustre-MDT0000 complete [ 2315.638378] LustreError: 71741:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2322.602637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.695624] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2322.942625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.982644] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2326.051190] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2327.519982] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2328.050724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2328.085143] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2328.176164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2336.740477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2336.750293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.762107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2338.273435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2343.407522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2343.412208] Lustre: Skipped 2 previous similar messages [ 2348.514047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2348.517352] Lustre: Skipped 1 previous similar message [ 2350.048125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2350.210536] Lustre: server umount lustre-OST0000 complete [ 2358.754457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2358.761290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2358.774133] Lustre: Skipped 1 previous similar message [ 2358.779026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2358.783880] Lustre: Skipped 1 previous similar message [ 2361.103911] Lustre: server umount lustre-MDT0000 complete [ 2363.880215] LustreError: 68662:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.930484] LustreError: 68662:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 2364.806052] LustreError: 68650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052195 with bad export cookie 3636980787337094502 [ 2364.817729] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2365.537906] Lustre: server umount lustre-MDT0001 complete [ 2383.594329] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2386.605255] Key type lgssc unregistered [ 2387.080340] LNet: 73557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.089144] LNetError: 73557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.149188] LNet: Removed LNI 192.168.202.131@tcp [ 2389.037026] Key type .llcrypt unregistered [ 2389.038677] Key type ._llcrypt unregistered [ 2412.349986] Key type ._llcrypt registered [ 2412.351985] Key type .llcrypt registered [ 2412.456474] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2413.495709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2413.582916] alg: No test for adler32 (adler32-zlib) [ 2414.647577] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2414.918412] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2416.641146] Key type lgssc registered [ 2417.844963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2425.449073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2425.465179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2426.960454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.962713] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2437.097817] LustreError: 74963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.116548] LustreError: 74963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2441.151180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.563957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2446.244558] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2453.546695] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2459.583164] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2466.454836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2466.787359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.938664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2473.455642] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2481.294149] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2490.993940] Lustre: Failing over lustre-MDT0000 [ 2491.286767] Lustre: server umount lustre-MDT0000 complete [ 2491.375152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.380133] LustreError: 74960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.385208] Lustre: Skipped 1 previous similar message [ 2491.402615] LustreError: 74960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2494.515721] LustreError: 74958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.617023] LustreError: 74959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.636212] LustreError: 74959:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2500.494324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.609641] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2500.777433] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2500.982390] LustreError: 78051:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2500.993302] LustreError: 78051:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.131@tcp: cfg command failed: rc = -9 [ 2501.007161] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2501.007161] [ 2501.015313] LustreError: MGC192.168.202.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2501.027313] LustreError: 78039:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2501.034873] LustreError: 78039:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2501.043337] LustreError: 78039:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2501.047987] LustreError: 78039:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2501.075942] Lustre: server umount lustre-MDT0000 complete [ 2501.079403] LustreError: 78039:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2504.737312] LustreError: 77454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.744976] LustreError: 77454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2509.050236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.277629] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2509.616547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2509.689841] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2509.872027] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2514.650361] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2514.928687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2514.955380] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2514.986994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2525.159408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.174725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2525.204310] Lustre: Skipped 2 previous similar messages [ 2529.054560] Lustre: server umount lustre-OST0000 complete [ 2535.393682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.431588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2535.441991] Lustre: Skipped 2 previous similar messages [ 2539.640182] Lustre: server umount lustre-MDT0000 complete [ 2540.523717] LustreError: 74960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.544731] LustreError: 74960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2543.790499] LustreError: 74943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052374 with bad export cookie 7730138619249422166 [ 2543.803614] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2544.181868] Lustre: server umount lustre-MDT0001 complete [ 2553.150995] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2555.990180] Key type lgssc unregistered [ 2556.375140] LNet: 79829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2556.384205] LNetError: 79829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2557.419372] LNet: Removed LNI 192.168.202.131@tcp [ 2558.538499] Key type .llcrypt unregistered [ 2558.541493] Key type ._llcrypt unregistered [ 2576.568459] Key type ._llcrypt registered [ 2576.575185] Key type .llcrypt registered [ 2576.707297] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2577.078382] Key type .llcrypt unregistered [ 2577.082138] Key type ._llcrypt unregistered [ 2601.909604] Key type ._llcrypt registered [ 2601.913344] Key type .llcrypt registered [ 2602.160955] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2603.620730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2603.702697] alg: No test for adler32 (adler32-zlib) [ 2604.985274] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2605.441687] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2607.120529] Key type lgssc registered [ 2608.716717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2618.121293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2618.186811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2619.671692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.514990] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2629.093469] LustreError: 81728:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.111912] LustreError: 81728:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2631.168635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.788701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2635.189812] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2642.166339] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2647.783714] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2654.469326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2654.607094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.640581] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2663.868960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2671.576460] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2687.457940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2687.469847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2687.500262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2689.507118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2689.514234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.514798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2690.933896] Lustre: server umount lustre-OST0000 complete [ 2698.211069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2698.237870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.269158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2698.276165] Lustre: Skipped 1 previous similar message [ 2701.755856] Lustre: server umount lustre-MDT0000 complete [ 2704.866556] LustreError: 81728:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.879868] LustreError: 81728:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2705.523121] LustreError: 81709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052535 with bad export cookie 13965365742643387977 [ 2705.535467] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.545420] LustreError: 81709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2706.295400] Lustre: server umount lustre-MDT0001 complete [ 2717.889619] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2720.989851] Key type lgssc unregistered [ 2721.363385] LNet: 85290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2721.374395] LNetError: 85290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2721.395141] LNet: Removed LNI 192.168.202.131@tcp [ 2722.513039] Key type .llcrypt unregistered [ 2722.514820] Key type ._llcrypt unregistered [ 2747.676471] Key type ._llcrypt registered [ 2747.678235] Key type .llcrypt registered [ 2747.778857] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2749.006568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2749.021930] alg: No test for adler32 (adler32-zlib) [ 2750.084307] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2750.303676] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2751.976227] Key type lgssc registered [ 2753.109043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2761.151797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2761.165437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.655361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.280115] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2771.938761] LustreError: 86704:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.963154] LustreError: 86704:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2775.706652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.116576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2781.243988] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2789.129627] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2795.125326] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2802.375333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2802.799302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.897537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2809.842707] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2817.264200] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2826.735524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.737324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2826.739054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2826.757307] Lustre: Skipped 1 previous similar message [ 2828.390808] Lustre: server umount lustre-OST0000 complete [ 2832.354452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2832.369276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2832.407016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2832.411355] Lustre: Skipped 1 previous similar message [ 2836.964171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2836.967129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.605072] Lustre: server umount lustre-MDT0000 complete [ 2842.085835] LustreError: 86700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.119912] LustreError: 86700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2842.329543] LustreError: 86683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052672 with bad export cookie 5826110945854223667 [ 2842.331520] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2842.353825] LustreError: 86683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2842.798484] Lustre: server umount lustre-MDT0001 complete [ 2856.244740] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2858.754975] Key type lgssc unregistered [ 2859.021689] LNet: 90233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2859.025509] LNetError: 90233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2859.041803] LNet: Removed LNI 192.168.202.131@tcp [ 2859.687975] Key type .llcrypt unregistered [ 2859.693553] Key type ._llcrypt unregistered [ 2882.426820] Key type ._llcrypt registered [ 2882.429611] Key type .llcrypt registered [ 2882.558951] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2883.961453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2884.048073] alg: No test for adler32 (adler32-zlib) [ 2885.148544] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2885.522150] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2887.176309] Key type lgssc registered [ 2888.724732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2898.527279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2898.565445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2900.227479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2905.061514] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2910.692247] LustreError: 91624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.701164] LustreError: 91624:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2914.005104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.235754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2919.669951] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2928.428680] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2935.040556] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2942.283979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2942.732844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2944.839675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2950.748116] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2957.494942] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2973.164435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2973.181142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.212096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2974.689312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2974.691169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2974.711145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.757420] Lustre: Skipped 1 previous similar message [ 2979.056122] Lustre: server umount lustre-OST0000 complete [ 2984.934600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.942765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2984.948107] Lustre: Skipped 1 previous similar message [ 2984.967826] Lustre: Skipped 1 previous similar message [ 2988.768450] Lustre: server umount lustre-MDT0000 complete [ 2990.051428] LustreError: 91624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.083504] LustreError: 91624:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2993.067239] LustreError: 91604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052823 with bad export cookie 16606732782894016636 [ 2993.097990] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2993.101921] LustreError: 91604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2993.483806] Lustre: server umount lustre-MDT0001 complete [ 3003.525774] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3006.754278] Key type lgssc unregistered [ 3007.145695] LNet: 95178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3007.162307] LNetError: 95178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3007.184053] LNet: Removed LNI 192.168.202.131@tcp [ 3008.090688] Key type .llcrypt unregistered [ 3008.092892] Key type ._llcrypt unregistered [ 3033.920708] Key type ._llcrypt registered [ 3033.923760] Key type .llcrypt registered [ 3034.036744] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3035.781407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3035.815921] alg: No test for adler32 (adler32-zlib) [ 3036.965703] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 3037.379318] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3039.109114] Key type lgssc registered [ 3040.831800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3049.117604] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3049.131289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3064.672417] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3069.793366] LustreError: 96561:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 3096.480713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3103.848359] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3111.673814] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3118.262167] Lustre: server umount lustre-OST0000 complete [ 3126.620449] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 11:09:15 (1789052955) [ 3132.438806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.905098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3138.045949] LustreError: 98202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3138.070153] LustreError: 98202:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3138.282279] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3143.140893] LustreError: 98201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3144.162820] LustreError: 98202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.615380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.440817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3154.689412] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3162.015426] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3167.013809] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3174.786935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3174.996082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3179.134245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3181.811731] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3189.610056] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3205.103356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.109245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3205.138488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3205.160591] Lustre: Skipped 1 previous similar message [ 3210.225922] LustreError: 100022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3210.247272] LustreError: 100022:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3210.405956] Lustre: server umount lustre-OST0000 complete [ 3215.332088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.374354] Lustre: Skipped 1 previous similar message [ 3215.398715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3215.411738] Lustre: Skipped 1 previous similar message [ 3220.453331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3220.465973] Lustre: Skipped 1 previous similar message [ 3221.945763] Lustre: server umount lustre-MDT0000 complete [ 3225.572634] LustreError: 100693:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.583762] LustreError: 100693:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3225.875521] LustreError: 98182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053056 with bad export cookie 8588260325007677460 [ 3225.884284] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3225.889515] LustreError: 98182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3226.321956] Lustre: server umount lustre-MDT0001 complete [ 3239.349471] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3241.869247] Key type lgssc unregistered [ 3242.197720] LNet: 101893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3242.213115] LNetError: 101893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3242.237676] LNet: Removed LNI 192.168.202.131@tcp [ 3243.314854] Key type .llcrypt unregistered [ 3243.322675] Key type ._llcrypt unregistered [ 3290.857747] Key type ._llcrypt registered [ 3290.859489] Key type .llcrypt registered [ 3290.968732] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3292.372108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3292.520425] alg: No test for adler32 (adler32-zlib) [ 3293.676902] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 3293.991855] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3295.712561] Key type lgssc registered [ 3297.509777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3304.648775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3304.671928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.084611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.641528] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3316.707844] LustreError: 104077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.729172] LustreError: 104077:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3318.843269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.160902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3324.442939] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3333.816283] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3341.278958] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3348.729533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3349.119703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.183544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3358.546918] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3368.386721] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3380.198159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3380.207100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3380.225985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3380.229807] Lustre: Skipped 1 previous similar message [ 3380.249155] Lustre: Skipped 1 previous similar message [ 3384.451981] Lustre: server umount lustre-OST0000 complete [ 3390.435362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3390.444695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3390.463297] Lustre: Skipped 1 previous similar message [ 3390.480366] Lustre: Skipped 1 previous similar message [ 3395.554747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3395.562149] Lustre: Skipped 1 previous similar message [ 3400.680184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3400.689176] Lustre: Skipped 1 previous similar message [ 3403.232106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3403.649930] Lustre: server umount lustre-MDT0000 complete [ 3405.793942] LustreError: 104074:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3405.840448] LustreError: 104074:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3407.698504] LustreError: 104814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053237 with bad export cookie 12011058864086955185 [ 3407.700202] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3407.703950] LustreError: 104814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3407.914791] Lustre: server umount lustre-MDT0001 complete [ 3419.603504] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3423.351723] Key type lgssc unregistered [ 3423.672862] LNet: 107615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3423.684897] LNetError: 107615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3423.705981] LNet: Removed LNI 192.168.202.131@tcp [ 3424.652810] Key type .llcrypt unregistered [ 3424.654409] Key type ._llcrypt unregistered [ 3445.879675] Key type ._llcrypt registered [ 3445.883525] Key type .llcrypt registered [ 3445.988732] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3447.805835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3447.846121] alg: No test for adler32 (adler32-zlib) [ 3448.967601] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 3449.211022] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3450.984434] Key type lgssc registered [ 3452.218723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3460.860292] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3460.885391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3476.515620] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3481.632372] LustreError: 108855:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 3508.320894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3514.040629] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3521.708304] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3529.869619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.113990] LustreError: MGC192.168.202.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3530.124402] LustreError: 109879:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3530.130449] LustreError: 109879:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3530.139554] LustreError: 109879:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3530.272792] Lustre: server umount lustre-MDT0000 complete [ 3530.275443] LustreError: 109879:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3551.712167] Lustre: 108192:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053365/real 1789053365] req@000000001e72ab07 x1875958331084032/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789053381 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3551.751768] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3551.772375] LustreError: 108867:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3572.422953] Lustre: server umount lustre-OST0000 complete [ 3584.736231] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 3599.285702] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3641.678553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.923401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3641.988071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3642.128276] Lustre: lustre-MDT0000: new disk, initializing [ 3642.220852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.252649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3648.941482] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3660.643312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3660.833610] Lustre: 112689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3660.923701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3660.927731] Lustre: Skipped 1 previous similar message [ 3661.062841] Lustre: lustre-MDT0001: new disk, initializing [ 3661.207378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3661.277247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3661.302751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3666.164515] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3674.416808] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3680.407922] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3689.014303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3689.506356] Lustre: lustre-OST0000: new disk, initializing [ 3689.509821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3689.515043] Lustre: 114655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3689.605438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3695.663636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3695.673861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3695.883577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3696.930702] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3705.708861] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3712.731057] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3713.251235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3719.430954] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3719.677656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3726.823118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3726.826976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.840517] LustreError: Skipped 1 previous similar message [ 3726.841212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3726.876061] Lustre: Skipped 1 previous similar message [ 3728.844575] Lustre: server umount lustre-OST0000 complete [ 3732.964040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3732.971492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3732.987522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3732.989834] Lustre: Skipped 1 previous similar message [ 3737.059396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3737.067049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3738.502774] Lustre: server umount lustre-MDT0000 complete [ 3742.413373] LustreError: 112679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053572 with bad export cookie 7849532496609937723 [ 3742.423131] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3742.433848] LustreError: 112679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3742.924193] Lustre: server umount lustre-MDT0001 complete [ 3751.032888] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 11:19:40 (1789053580) [ 3755.088421] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3772.977088] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 3784.548826] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3819.075757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.332574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3819.403942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3819.512399] Lustre: lustre-MDT0000: new disk, initializing [ 3819.617493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3819.642812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3825.200225] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3836.856830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3836.928851] Lustre: 119556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3836.956165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3836.959098] Lustre: Skipped 1 previous similar message [ 3836.999853] Lustre: lustre-MDT0001: new disk, initializing [ 3837.113589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3837.167294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3837.181521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3841.250925] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3851.435126] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3857.193688] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3863.783726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3864.151262] Lustre: lustre-OST0000: new disk, initializing [ 3864.155992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3864.166414] Lustre: 121524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3864.260746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3870.041157] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3872.915110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3872.928389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3873.005481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3879.070626] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3882.633335] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3882.871065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3887.366409] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3887.577350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3893.734330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3893.764026] Lustre: Skipped 1 previous similar message [ 3893.770154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3893.773386] Lustre: Skipped 1 previous similar message [ 3896.305948] Lustre: server umount lustre-OST0000 complete [ 3903.969473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3903.973606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3903.984265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3904.003687] Lustre: Skipped 1 previous similar message [ 3904.020029] Lustre: Skipped 1 previous similar message [ 3907.357975] Lustre: server umount lustre-MDT0000 complete [ 3909.090431] LustreError: 120719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3909.099875] LustreError: 120719:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3910.736311] LustreError: 120351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053741 with bad export cookie 7849532496609938612 [ 3910.741857] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3910.756143] LustreError: 120351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3911.040655] Lustre: server umount lustre-MDT0001 complete [ 3919.101405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3919.807549] LustreError: 123739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3919.856429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3924.482734] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3924.960988] LustreError: 123740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3930.081637] LustreError: 123739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3932.100684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3936.365695] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3943.814918] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3949.643806] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3956.222070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3962.957453] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3970.362895] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3976.445433] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3980.259170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3980.259564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.271132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.275024] Lustre: Skipped 1 previous similar message [ 3980.290694] Lustre: Skipped 1 previous similar message [ 3986.206216] Lustre: server umount lustre-OST0000 complete [ 3988.963176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4000.739296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4000.743605] Lustre: Skipped 8 previous similar messages [ 4003.296126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4003.630331] Lustre: server umount lustre-MDT0000 complete [ 4005.860174] LustreError: 123740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4005.889494] LustreError: 123740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4008.978648] LustreError: 123721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053839 with bad export cookie 7849532496609939494 [ 4009.002235] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.025856] LustreError: 123721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4009.945239] Lustre: server umount lustre-MDT0001 complete [ 4023.079637] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4026.288382] Key type lgssc unregistered [ 4026.774069] LNet: 127380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4026.786492] LNetError: 127380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4027.817897] LNet: Removed LNI 192.168.202.131@tcp [ 4028.682761] Key type .llcrypt unregistered [ 4028.685681] Key type ._llcrypt unregistered [ 4045.536870] Key type ._llcrypt registered [ 4045.539387] Key type .llcrypt registered [ 4045.675595] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 4061.743148] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4062.733736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4062.821304] alg: No test for adler32 (adler32-zlib) [ 4063.936444] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 4064.253888] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4065.920356] Key type lgssc registered [ 4067.154305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4101.654554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4101.678661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4103.048221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4103.096778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4103.217366] Lustre: lustre-MDT0000: new disk, initializing [ 4103.268051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.300487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4108.069647] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4118.836647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4118.997346] Lustre: 130532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4119.074864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4119.079537] Lustre: Skipped 1 previous similar message [ 4119.153505] Lustre: lustre-MDT0001: new disk, initializing [ 4119.260678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4119.320319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4119.340277] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4125.108808] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4135.317414] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4140.766657] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4148.100707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4148.414452] Lustre: lustre-OST0000: new disk, initializing [ 4148.418955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4148.428129] Lustre: 132499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4148.511842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4153.923694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4153.930649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4154.055516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4154.992761] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4166.767185] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4173.899988] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4174.204563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4179.552445] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4179.757733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4185.065619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.077262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4185.102314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4185.111663] LustreError: Skipped 1 previous similar message [ 4185.129706] Lustre: Skipped 1 previous similar message [ 4190.186559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4190.198904] Lustre: Skipped 1 previous similar message [ 4195.300718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4195.311406] Lustre: Skipped 1 previous similar message [ 4196.320139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4196.414953] Lustre: server umount lustre-OST0000 complete [ 4200.419856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.431608] Lustre: Skipped 1 previous similar message [ 4200.437217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4200.444671] Lustre: Skipped 1 previous similar message [ 4205.538531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.547152] Lustre: Skipped 1 previous similar message [ 4205.927369] Lustre: server umount lustre-MDT0000 complete [ 4209.610072] LustreError: 131326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054039 with bad export cookie 5259908849113326272 [ 4209.610691] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4209.618629] LustreError: 131326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4210.090335] Lustre: server umount lustre-MDT0001 complete [ 4218.617869] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 11:27:27 (1789054047) [ 4226.917727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.949314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.006262] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4233.197337] LustreError: 134912:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.217964] LustreError: 134912:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4238.321736] LustreError: 134913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.196367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4241.447083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4246.624566] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4255.625899] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4262.053255] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4263.909678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.914662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4263.920888] Lustre: Skipped 2 previous similar messages [ 4270.048808] Lustre: server umount lustre-MDT0000 complete [ 4271.075215] LustreError: 134913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4271.089602] LustreError: 134913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4273.791777] LustreError: 134894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054104 with bad export cookie 5259908849113327154 [ 4273.796863] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4273.802029] LustreError: 134894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4274.111808] Lustre: server umount lustre-MDT0001 complete [ 4282.391892] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 11:28:31 (1789054111) [ 4288.617744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4304.160709] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4309.280718] LustreError: 137328:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 4335.008519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.361475] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4349.697319] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4352.493143] Lustre: server umount lustre-OST0000 complete [ 4361.145887] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 11:29:50 (1789054190) [ 4368.494157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4369.056133] LustreError: 138775:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4369.231674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4374.158853] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4374.514782] LustreError: 138776:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.077075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4388.816921] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4397.927329] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4404.995862] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4412.900375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4413.333133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4413.345674] Lustre: Skipped 1 previous similar message [ 4421.447698] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4430.366155] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4444.138278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4444.148667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4444.173126] Lustre: Skipped 1 previous similar message [ 4444.193446] Lustre: Skipped 5 previous similar messages [ 4447.874164] Lustre: server umount lustre-MDT0000 complete [ 4449.250491] LustreError: 138775:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4449.275499] LustreError: 138775:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4451.697108] LustreError: 138755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054281 with bad export cookie 5259908849113327819 [ 4451.714094] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4451.721832] LustreError: 138755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4452.165146] Lustre: server umount lustre-MDT0001 complete [ 4467.113466] Lustre: server umount lustre-OST0000 complete [ 4477.719428] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 11:31:46 (1789054306) [ 4486.537676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4486.942082] LustreError: 142429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4487.016230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4492.218761] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4501.546105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.224653] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4514.079677] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4521.695323] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4529.356284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4532.048193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4538.139552] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4546.731751] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4552.625834] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4552.870981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4558.871635] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4559.143323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4562.916366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.918359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4562.936264] Lustre: Skipped 1 previous similar message [ 4562.950740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4562.973299] Lustre: Skipped 1 previous similar message [ 4567.670424] Lustre: server umount lustre-OST0000 complete [ 4573.160502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.176155] Lustre: Skipped 2 previous similar messages [ 4578.048437] Lustre: server umount lustre-MDT0000 complete [ 4578.277542] LustreError: 143160:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4578.295084] LustreError: 143160:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4581.778328] LustreError: 142411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054412 with bad export cookie 5259908849113328890 [ 4581.788683] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4581.799942] LustreError: 142411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4582.551330] Lustre: server umount lustre-MDT0001 complete [ 4591.276391] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 11:33:40 (1789054420) [ 4597.313621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4612.832736] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4617.955323] LustreError: 146591:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 4643.680291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4643.687897] Lustre: Skipped 2 previous similar messages [ 4651.109375] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4658.488816] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4665.890400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4666.249875] LustreError: 146616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4666.381043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4670.526984] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4680.282284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4686.420958] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4693.552944] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4699.081981] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4704.676464] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4705.074246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4711.984395] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4712.410162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4716.514620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4716.516169] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4716.534289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4716.534299] Lustre: Skipped 2 previous similar messages [ 4716.572432] Lustre: Skipped 2 previous similar messages [ 4721.775299] Lustre: server umount lustre-MDT0000 complete [ 4726.404325] LustreError: 146598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054556 with bad export cookie 5259908849113329912 [ 4726.406482] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4726.414578] LustreError: 146598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4726.624357] Lustre: server umount lustre-MDT0001 complete [ 4741.695316] Lustre: server umount lustre-OST0000 complete [ 4752.359442] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 11:36:20 (1789054580) [ 4760.454414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4776.224631] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4781.344480] LustreError: 150788:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 4813.385343] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4820.697604] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4828.069623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4828.681255] LustreError: 150813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4828.708630] LustreError: 150813:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4828.936185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4834.426608] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4843.428823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.252724] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4856.767862] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4862.781320] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4871.492935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4871.774952] Lustre: lustre-OST0001: new disk, initializing [ 4871.779928] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4871.793755] Lustre: 153657:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4878.943909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4878.956792] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4879.109712] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4879.154706] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4888.578228] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4894.737074] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4895.183190] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4901.407696] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4901.732470] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4904.934133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4904.938231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4904.942929] Lustre: Skipped 1 previous similar message [ 4904.952641] Lustre: Skipped 3 previous similar messages [ 4919.264943] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4919.383741] Lustre: server umount lustre-OST0000 complete [ 4925.423257] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4929.484965] Lustre: server umount lustre-OST0001 complete [ 4939.675656] Lustre: server umount lustre-MDT0000 complete [ 4944.363472] LustreError: 150795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054774 with bad export cookie 5259908849113330864 [ 4944.377976] LustreError: 150795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4944.379397] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4944.675229] Lustre: server umount lustre-MDT0001 complete [ 4971.306909] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 11:39:59 (1789054799) [ 4973.967359] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4976.569230] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4979.482710] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 11:40:07 (1789054807) [ 4988.090321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4988.605594] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4988.704449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4989.173961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.176591] Lustre: Skipped 6 previous similar messages [ 4996.192636] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5006.135030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5006.252112] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5006.284535] Lustre: 157222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5011.290930] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5018.981536] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5025.556558] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5033.116863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5033.451750] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5039.344685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5040.831913] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5049.064940] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5053.938447] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5054.215709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5059.148102] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5059.385751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5064.694300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5064.708231] LustreError: Skipped 1 previous similar message [ 5064.709527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5064.726433] Lustre: Skipped 4 previous similar messages [ 5068.163474] Lustre: server umount lustre-OST0000 complete [ 5091.482038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5100.007347] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5105.198116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.215123] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5108.657978] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5114.861026] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5115.319586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5115.492874] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.533957] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5121.038408] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5121.249624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5126.893967] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa075e0de7800.ost_server_uuid 50 [ 5128.724330] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa075e0de7800.ost_server_uuid in FULL state after 0 sec [ 5128.992982] LustreError: 157228:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.31@tcp) returned error from blocking AST (req@00000000cdaa58c7 x1875958976230784 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000000b2c2ec/0x48fef33ae6557e0a lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.31@tcp remote: 0xf7f1bba910bf5cd3 expref: 5 pid: 157228 timeout: 5228 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5129.047563] LustreError: lustre-MDT0000: A client on nid 192.168.202.31@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5129.081225] LustreError: 157218:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.202.31@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000000b2c2ec/0x48fef33ae6557e0a lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.31@tcp remote: 0xf7f1bba910bf5cd3 expref: 6 pid: 157228 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5134.831637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5134.840306] LustreError: Skipped 1 previous similar message [ 5140.115569] Lustre: server umount lustre-OST0000 complete [ 5150.927839] Lustre: server umount lustre-MDT0000 complete [ 5151.714896] LustreError: 157233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.748247] LustreError: 157233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 5155.259128] LustreError: 157213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054985 with bad export cookie 5259908849113332047 [ 5155.282746] LustreError: 157213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5155.283826] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5162.178864] Lustre: server umount lustre-MDT0001 complete [ 5172.989958] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5176.242979] Key type lgssc unregistered [ 5176.785194] LNet: 163343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5176.809645] LNetError: 163343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5176.836621] LNet: Removed LNI 192.168.202.131@tcp [ 5177.876530] Key type .llcrypt unregistered [ 5177.878767] Key type ._llcrypt unregistered [ 5205.671292] Key type ._llcrypt registered [ 5205.678046] Key type .llcrypt registered [ 5205.864186] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5207.175500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5207.198702] alg: No test for adler32 (adler32-zlib) [ 5208.482571] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 5208.906913] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5210.627089] Key type lgssc registered [ 5211.950704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5221.076403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5221.132564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.676939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5227.415626] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5228.002762] LustreError: 164757:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.034122] LustreError: 164757:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5232.097745] LustreError: 164758:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.084793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5237.220310] LustreError: 164757:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.825422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5243.701501] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5251.818318] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5258.029744] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5264.811372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5265.332220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5267.404141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5272.524658] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5282.909976] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5290.808244] Lustre: Failing over lustre-MDT0000 [ 5291.008261] Lustre: server umount lustre-MDT0000 complete [ 5291.151627] LustreError: 164753:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5293.044963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.077715] Lustre: Skipped 2 previous similar messages [ 5298.145949] LustreError: 164758:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5298.172440] LustreError: 164758:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5304.254094] LustreError: 164739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.31@tcp arrived at 1789055134 with bad export cookie 18390792286162540182 [ 5304.267660] LustreError: 164739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5308.388597] LustreError: 165684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5308.407061] LustreError: 165684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5309.414528] Lustre: 164067:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055123/real 1789055123] req@00000000c8a9846d x1875960176602368/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789055139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5309.463833] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5311.943196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.071209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5320.123381] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5325.285525] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5325.290365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5325.577473] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5328.417574] Lustre: lustre-MDT0000: Denying connection for new client 9f637b7f-ea2e-4937-8119-899c4e541f24 (at 192.168.202.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5339.106334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5339.108838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.135992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5340.645647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5340.649184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5342.916331] Lustre: server umount lustre-OST0000 complete [ 5347.111299] LustreError: 168491:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5347.125887] Lustre: 167838:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5347.137447] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5347.153269] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5347.164214] LustreError: 167838:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000115d52fe x1875960176623616/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5347.181555] LustreError: 167838:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5347.194124] LustreError: 167838:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5347.214547] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5348.159342] Lustre: server umount lustre-MDT0000 complete [ 5350.900754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5350.916120] LustreError: 168248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5350.953233] LustreError: 168248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 5353.117964] LustreError: 164738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055183 with bad export cookie 18390792286162540280 [ 5353.128483] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5353.135674] LustreError: 164738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5353.137243] LustreError: 164063:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5359.772755] Lustre: server umount lustre-MDT0001 complete [ 5372.456769] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5375.245057] Key type lgssc unregistered [ 5375.579331] LNet: 169245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5375.590266] LNetError: 169245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5375.617177] LNet: Removed LNI 192.168.202.131@tcp [ 5376.370540] Key type .llcrypt unregistered [ 5376.372677] Key type ._llcrypt unregistered [ 5399.075405] Key type ._llcrypt registered [ 5399.077661] Key type .llcrypt registered [ 5399.185870] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5400.722033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5400.741883] alg: No test for adler32 (adler32-zlib) [ 5402.049769] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 5402.407738] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5404.256342] Key type lgssc registered [ 5406.146879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5417.007552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5417.060304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5418.734994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5424.168485] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5428.199937] LustreError: 170653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5428.249208] LustreError: 170653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5432.828212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5433.307725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5438.182982] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5447.310702] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5454.998241] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5464.156265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5464.636737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5472.157785] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5480.497423] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5488.266971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5503.980450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.984465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5503.997553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5504.023654] LustreError: Skipped 1 previous similar message [ 5505.350759] Lustre: server umount lustre-OST0000 complete [ 5510.114121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5510.123339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.142448] Lustre: Skipped 1 previous similar message [ 5510.151287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5510.171840] Lustre: Skipped 1 previous similar message [ 5514.212759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5514.213860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5515.778671] Lustre: server umount lustre-MDT0000 complete [ 5519.332770] LustreError: 170654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5519.353687] LustreError: 170654:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5520.208472] LustreError: 170633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055350 with bad export cookie 12605584339407237976 [ 5520.212863] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5520.221722] LustreError: 170633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5520.768727] Lustre: server umount lustre-MDT0001 complete [ 5535.479538] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5538.624199] Key type lgssc unregistered [ 5538.879859] LNet: 174211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5538.884525] LNetError: 174211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5538.937441] LNet: Removed LNI 192.168.202.131@tcp [ 5539.906283] Key type .llcrypt unregistered [ 5539.909401] Key type ._llcrypt unregistered [ 5575.981712] Key type ._llcrypt registered [ 5575.986509] Key type .llcrypt registered [ 5576.124463] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5577.435175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5577.537618] alg: No test for adler32 (adler32-zlib) [ 5578.724930] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 5578.947695] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5580.648303] Key type lgssc registered [ 5581.875720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5591.627270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5591.642787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5593.263332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.227231] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5602.797113] LustreError: 175912:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5602.813048] LustreError: 175912:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5607.920748] LustreError: 175911:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5609.360082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5609.914602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5615.884907] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5626.060861] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5633.477851] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5643.054706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5643.763570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5652.587307] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5660.780651] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5663.215954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5673.500451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5673.516618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.558574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5673.582675] Lustre: Skipped 1 previous similar message [ 5678.481836] Lustre: server umount lustre-OST0000 complete [ 5683.685562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5683.691866] Lustre: Skipped 1 previous similar message [ 5683.695646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5683.697879] Lustre: Skipped 1 previous similar message [ 5688.803163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5688.822402] Lustre: Skipped 1 previous similar message [ 5689.143471] Lustre: server umount lustre-MDT0000 complete [ 5692.907668] LustreError: 176846:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5692.937925] LustreError: 176846:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5695.092598] LustreError: 175891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055525 with bad export cookie 13659549680607608887 [ 5695.097325] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5695.098619] LustreError: 175891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5695.496353] Lustre: server umount lustre-MDT0001 complete [ 5709.877864] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5713.036963] Key type lgssc unregistered [ 5713.403752] LNet: 179447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5713.407283] LNetError: 179447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5713.424071] LNet: Removed LNI 192.168.202.131@tcp [ 5714.157637] Key type .llcrypt unregistered [ 5714.158755] Key type ._llcrypt unregistered [ 5741.181244] Key type ._llcrypt registered [ 5741.184688] Key type .llcrypt registered [ 5741.325952] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5742.701263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5742.761938] alg: No test for adler32 (adler32-zlib) [ 5743.978614] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 5744.261176] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5746.000674] Key type lgssc registered [ 5747.507493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5759.360781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5759.372878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5760.777404] Lustre: *** cfs_fail_loc=135, val=0*** [ 5760.882357] LustreError: 180968:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5760.890137] LustreError: 180968:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.131@tcp: cfg command failed: rc = -2 [ 5760.900725] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5760.900725] [ 5760.914170] LustreError: MGC192.168.202.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5760.931260] LustreError: 180929:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5760.945707] LustreError: 180929:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5760.963572] LustreError: 180929:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5760.975655] LustreError: 180929:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5761.099367] Lustre: server umount lustre-MDT0000 complete [ 5761.105051] LustreError: 180929:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5768.871154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5769.353880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5774.973425] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5778.921445] LustreError: 181340:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.935542] LustreError: 181340:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5784.042691] LustreError: 181339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5785.483278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5785.852582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5790.498480] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5798.417593] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5806.119558] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5811.617649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5811.637243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5811.647067] Lustre: Skipped 1 previous similar message [ 5811.668207] Lustre: Skipped 1 previous similar message [ 5814.971916] Lustre: server umount lustre-MDT0000 complete [ 5816.804776] LustreError: 182069:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5816.820664] LustreError: 182069:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5819.793983] LustreError: 181319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055650 with bad export cookie 16200388482144374643 [ 5819.805866] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5820.619811] Lustre: server umount lustre-MDT0001 complete [ 5832.195403] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5835.270906] Key type lgssc unregistered [ 5835.734055] LNet: 183664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5835.748621] LNetError: 183664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5835.787068] LNet: Removed LNI 192.168.202.131@tcp [ 5836.634744] Key type .llcrypt unregistered [ 5836.636650] Key type ._llcrypt unregistered [ 5859.307673] Key type ._llcrypt registered [ 5859.310412] Key type .llcrypt registered [ 5859.391646] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5859.770419] Key type .llcrypt unregistered [ 5859.773671] Key type ._llcrypt unregistered [ 5878.327119] Key type ._llcrypt registered [ 5878.331456] Key type .llcrypt registered [ 5878.489406] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5880.162063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5880.238614] alg: No test for adler32 (adler32-zlib) [ 5881.375910] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 5881.694613] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5883.401950] Key type lgssc registered [ 5884.693565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5892.449960] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5892.475069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5908.000657] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5913.122247] LustreError: 185788:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 5939.881235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5948.132675] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5956.706926] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5965.226407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.735578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5965.836152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5970.921504] LustreError: 185813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.943415] LustreError: 185813:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5971.069239] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5973.988896] LustreError: 186825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5974.027972] LustreError: 186825:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5979.124356] LustreError: 186825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5979.168584] LustreError: 186825:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5980.693244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5981.235038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5986.916835] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5995.579281] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6001.808243] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6004.940447] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6016.997572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6016.997615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6017.011118] LustreError: Skipped 1 previous similar message [ 6017.014187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6017.030632] Lustre: Skipped 1 previous similar message [ 6022.114741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6022.135278] Lustre: Skipped 2 previous similar messages [ 6022.379181] Lustre: server umount lustre-OST0000 complete [ 6027.242992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6027.245934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6027.249199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6027.251209] Lustre: Skipped 1 previous similar message [ 6027.277100] Lustre: Skipped 1 previous similar message [ 6032.358498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6032.367736] Lustre: Skipped 1 previous similar message [ 6033.469966] Lustre: server umount lustre-MDT0000 complete [ 6037.478918] LustreError: 189003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6037.513917] LustreError: 189003:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6038.479927] LustreError: 185794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055868 with bad export cookie 13336093720840832848 [ 6038.491251] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6039.305452] Lustre: server umount lustre-MDT0001 complete [ 6053.768226] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 6059.405960] Key type lgssc unregistered [ 6060.262614] LNet: 189756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6060.278108] LNetError: 189756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6061.366111] LNet: Removed LNI 192.168.202.131@tcp [ 6063.161340] Key type .llcrypt unregistered [ 6063.162919] Key type ._llcrypt unregistered [ 6096.271389] Key type ._llcrypt registered [ 6096.273321] Key type .llcrypt registered [ 6096.422113] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 6098.769752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6099.029881] alg: No test for adler32 (adler32-zlib) [ 6100.559507] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 6100.981974] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 6102.800271] Key type lgssc registered [ 6104.217559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6114.040233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6114.055531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6115.625237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6121.541429] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6125.030680] LustreError: 191170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6125.061730] LustreError: 191170:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6130.148905] LustreError: 191171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6131.045284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6131.398247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6136.367710] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6144.319558] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6150.498517] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6158.114632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6158.667404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6160.698758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6166.108938] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6174.734431] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6179.386766] Lustre: Failing over lustre-MDT0000 [ 6179.638804] Lustre: server umount lustre-MDT0000 complete [ 6181.351470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6181.359730] LustreError: 191898:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6181.362949] Lustre: Skipped 2 previous similar messages [ 6181.386649] LustreError: 191898:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6186.472793] LustreError: 191170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6186.500109] LustreError: 191170:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6191.599951] LustreError: 191171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6191.632949] LustreError: 191171:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6197.729101] Lustre: 190478:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789056011/real 1789056011] req@0000000028d6760e x1875961111931264/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789056027 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6197.740633] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6200.193501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6201.984613] LustreError: 191167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6202.022431] LustreError: 191167:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 6208.506284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6208.566321] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6212.146492] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6213.615740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6213.653086] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6213.739026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6213.859895] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6218.787466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6226.670437] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6234.594551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6234.606080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6234.636983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6237.665429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6237.670806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6237.703766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6239.212077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6239.220165] Lustre: Skipped 1 previous similar message [ 6241.003714] Lustre: server umount lustre-OST0000 complete [ 6249.443271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6249.443562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6249.466082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6249.474305] Lustre: Skipped 1 previous similar message [ 6254.566822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6254.580341] Lustre: Skipped 1 previous similar message [ 6261.218744] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6261.840765] Lustre: server umount lustre-MDT0000 complete [ 6263.790861] LustreError: 191170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6263.810431] LustreError: 191170:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 6266.353852] LustreError: 191906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056096 with bad export cookie 4466159728174344060 [ 6266.368053] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6266.369860] LustreError: 191906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6266.966173] Lustre: server umount lustre-MDT0001 complete [ 6278.859252] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 6281.868548] Key type lgssc unregistered [ 6282.157740] LNet: 196073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6282.165797] LNetError: 196073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6282.197337] LNet: Removed LNI 192.168.202.131@tcp [ 6283.180135] Key type .llcrypt unregistered [ 6283.183214] Key type ._llcrypt unregistered [ 6310.864906] Key type ._llcrypt registered [ 6310.868266] Key type .llcrypt registered [ 6311.013410] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 6312.590181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6312.838990] alg: No test for adler32 (adler32-zlib) [ 6313.991162] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 6314.266978] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 6316.040545] Key type lgssc registered [ 6317.498591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6325.321812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6325.341098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6326.922428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6331.958746] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6337.515792] LustreError: 197483:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6337.528283] LustreError: 197483:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6341.917834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6342.436454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6347.450494] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6357.283133] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6365.336648] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6374.175949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6374.463665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6380.008638] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6389.134553] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6396.978820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6402.179987] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6405.572985] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6423.486576] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6437.858831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6437.865395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6437.878653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6437.885774] Lustre: Skipped 1 previous similar message [ 6437.895853] Lustre: Skipped 1 previous similar message [ 6442.979837] LustreError: 199307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6443.148619] Lustre: server umount lustre-OST0000 complete [ 6448.097727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6448.130369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6453.218173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6453.222535] Lustre: Skipped 1 previous similar message [ 6454.042938] Lustre: server umount lustre-MDT0000 complete [ 6457.878531] LustreError: 197464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056288 with bad export cookie 13855865229154250755 [ 6457.883804] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6457.908085] LustreError: 197464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6458.371698] Lustre: server umount lustre-MDT0001 complete [ 6471.413233] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 6475.720787] Key type lgssc unregistered [ 6475.929731] LNet: 201401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6475.933555] LNetError: 201401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6475.946766] LNet: Removed LNI 192.168.202.131@tcp [ 6476.882588] Key type .llcrypt unregistered [ 6476.884970] Key type ._llcrypt unregistered [ 6500.253232] Key type ._llcrypt registered [ 6500.257392] Key type .llcrypt registered [ 6500.440752] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 6502.109436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6502.150974] alg: No test for adler32 (adler32-zlib) [ 6503.436980] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 6503.755548] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 6505.448319] Key type lgssc registered [ 6507.149363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6516.511223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6516.528036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6518.378550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6524.591514] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6528.491876] LustreError: 202663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6528.514564] LustreError: 202663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6533.612543] LustreError: 202664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6534.518156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.197736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6540.463115] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6547.889232] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6554.113694] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6561.775276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6562.391623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6563.479381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6569.244642] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6577.870055] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6583.914556] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6598.646987] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6603.687111] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6611.600988] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6625.249259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6625.258580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6625.286636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6626.279529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6626.283093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6626.290616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6629.443222] Lustre: server umount lustre-OST0000 complete [ 6636.515696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.548728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6636.558114] Lustre: Skipped 2 previous similar messages [ 6641.642675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6641.652888] Lustre: Skipped 1 previous similar message [ 6646.758456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6646.765054] Lustre: Skipped 1 previous similar message [ 6648.288468] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6648.559896] Lustre: server umount lustre-MDT0000 complete [ 6651.875862] LustreError: 202664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6651.903884] LustreError: 202664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6652.610474] LustreError: 202645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056482 with bad export cookie 10153841188742019769 [ 6652.622315] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6652.625679] LustreError: 202645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6653.202127] Lustre: server umount lustre-MDT0001 complete [ 6664.233495] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 6667.598249] Key type lgssc unregistered [ 6667.873663] LNet: 207589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6667.883732] LNetError: 207589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6667.896753] LNet: Removed LNI 192.168.202.131@tcp [ 6668.947896] Key type .llcrypt unregistered [ 6668.951115] Key type ._llcrypt unregistered [ 6696.205347] Key type ._llcrypt registered [ 6696.207821] Key type .llcrypt registered [ 6696.311577] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 6697.915106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6698.035261] alg: No test for adler32 (adler32-zlib) [ 6699.184153] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 6699.440573] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 6701.088213] Key type lgssc registered [ 6701.832325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6709.393752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6709.416590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6710.934394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6717.478949] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6721.506176] LustreError: 209000:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6721.516221] LustreError: 209000:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6726.631347] LustreError: 209001:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6729.595994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6730.523559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6736.893112] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6748.732212] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6755.739522] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6764.286870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6764.995393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6774.032145] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6782.008080] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6782.403865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6800.873959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6800.888612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.899069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6802.915253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6802.926897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6802.931538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6806.478822] Lustre: server umount lustre-OST0000 complete [ 6812.645480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6812.667064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6812.692844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6812.700157] Lustre: Skipped 1 previous similar message [ 6818.038757] Lustre: server umount lustre-MDT0000 complete [ 6818.281506] LustreError: 208997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6818.307740] LustreError: 208997:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6822.359020] LustreError: 208980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056652 with bad export cookie 7290263683533197751 [ 6822.370319] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6822.376190] LustreError: 208980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6823.005084] Lustre: server umount lustre-MDT0001 complete [ 6835.604723] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 6839.126065] Key type lgssc unregistered [ 6839.536268] LNet: 212688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6839.546841] LNetError: 212688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6839.584180] LNet: Removed LNI 192.168.202.131@tcp [ 6840.636081] Key type .llcrypt unregistered [ 6840.638755] Key type ._llcrypt unregistered [ 6864.413519] Key type ._llcrypt registered [ 6864.419667] Key type .llcrypt registered [ 6864.616187] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 6866.250564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6866.266074] alg: No test for adler32 (adler32-zlib) [ 6867.432889] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 6867.738558] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 6869.464310] Key type lgssc registered [ 6870.761861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6880.084411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6880.120787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6881.780773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6888.154935] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6890.980147] LustreError: 213951:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6891.030511] LustreError: 213951:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6896.099059] LustreError: 213952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6899.951715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.374994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6905.642476] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6913.630727] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6920.009476] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6928.935387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6929.653363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6938.227215] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6947.449758] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6952.192364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6970.850641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6970.858569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.882209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6972.904134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6972.907679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6972.911259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6972.916547] Lustre: Skipped 1 previous similar message [ 6975.975549] Lustre: server umount lustre-OST0000 complete [ 6982.123679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6982.135196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.153216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6988.272424] LustreError: 213951:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.315924] LustreError: 213951:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6988.628623] Lustre: server umount lustre-MDT0000 complete [ 6992.756173] LustreError: 213932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056823 with bad export cookie 7413523586148299249 [ 6992.768851] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6993.210756] Lustre: server umount lustre-MDT0001 complete [ 7004.918297] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 7008.441429] Key type lgssc unregistered [ 7008.824495] LNet: 217654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7008.839294] LNetError: 217654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7009.895637] LNet: Removed LNI 192.168.202.131@tcp [ 7010.870173] Key type .llcrypt unregistered [ 7010.872135] Key type ._llcrypt unregistered [ 7032.470578] Key type ._llcrypt registered [ 7032.474893] Key type .llcrypt registered [ 7032.692765] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 7034.280163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7034.524789] alg: No test for adler32 (adler32-zlib) [ 7035.619231] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 7035.871262] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 7037.528767] Key type lgssc registered [ 7038.884323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7047.555404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7047.582487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7049.213488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7054.672518] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7059.433901] LustreError: 218920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.478032] LustreError: 218920:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7064.545439] LustreError: 218921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7064.660046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7065.293314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7071.042257] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7079.203636] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7085.924636] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7094.305170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7094.711306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7101.681889] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7110.140769] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7119.285581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7131.559147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.615961] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7131.828650] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7139.593909] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7151.195713] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7154.350503] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7166.131364] Lustre: Permanently deactivating lustre-OST0001 [ 7166.157814] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7173.787810] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7181.316565] Lustre: Permanently reactivating lustre-OST0001 [ 7181.338305] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7181.342870] Lustre: Skipped 2 previous similar messages [ 7187.561616] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7187.590830] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7187.613630] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7187.637931] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7187.702549] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7198.184411] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7198.187360] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7198.188786] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7198.197133] LustreError: Skipped 1 previous similar message [ 7198.232547] Lustre: Skipped 2 previous similar messages [ 7201.689344] Lustre: server umount lustre-OST0001 complete [ 7203.305866] LustreError: 220742:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.333989] LustreError: 220742:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7206.372516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7206.378253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.400166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7206.403194] Lustre: Skipped 1 previous similar message [ 7207.394123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7207.408390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7208.432941] LustreError: 220744:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7208.451545] LustreError: 220744:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7213.551158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7213.565752] Lustre: Skipped 3 previous similar messages [ 7213.569660] LustreError: 220743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7213.600906] LustreError: 220743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7218.668481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7218.683180] Lustre: Skipped 1 previous similar message [ 7220.195752] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7220.581759] Lustre: server umount lustre-OST0000 complete [ 7228.899217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7228.913703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7228.917618] Lustre: Skipped 2 previous similar messages [ 7228.940038] Lustre: Skipped 1 previous similar message [ 7231.971895] Lustre: server umount lustre-MDT0000 complete [ 7234.025142] LustreError: 219648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7234.066096] LustreError: 219648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7236.712607] LustreError: 218900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057066 with bad export cookie 11205930061496673612 [ 7236.713825] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7236.728655] LustreError: 218900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7237.175819] Lustre: server umount lustre-MDT0001 complete [ 7250.656814] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 7253.475548] Key type lgssc unregistered [ 7253.715615] LNet: 224011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7253.720412] LNetError: 224011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7253.732774] LNet: Removed LNI 192.168.202.131@tcp [ 7254.555545] Key type .llcrypt unregistered [ 7254.557535] Key type ._llcrypt unregistered [ 7306.112995] Key type ._llcrypt registered [ 7306.118974] Key type .llcrypt registered [ 7306.314364] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 7308.127423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7308.205276] alg: No test for adler32 (adler32-zlib) [ 7309.445568] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 7309.661153] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 7311.388062] Key type lgssc registered [ 7312.680698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7322.051828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7322.075765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7323.327993] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7323.346662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7323.678290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7328.544649] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7339.807410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7340.038686] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7340.094566] Lustre: 226217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7340.352210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7346.552119] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7356.399589] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7364.317338] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7372.886222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7373.159298] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7373.375630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7381.242216] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7382.756319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7390.221603] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7394.457277] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7401.843902] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7410.043445] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7418.206434] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7433.191830] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7433.200588] Lustre: Skipped 1 previous similar message [ 7451.255942] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7451.262625] Lustre: Skipped 1 previous similar message [ 7488.731881] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7488.740510] Lustre: Skipped 4 previous similar messages [ 7561.185807] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7561.193710] Lustre: Skipped 8 previous similar messages [ 7571.945674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7571.956396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7571.978958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7572.456090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7572.476225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7572.497779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7572.500455] Lustre: Skipped 1 previous similar message [ 7575.944983] Lustre: server umount lustre-OST0000 complete [ 7581.154330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7581.165145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7581.178560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7586.735045] Lustre: server umount lustre-MDT0000 complete [ 7590.694145] LustreError: 228902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057420 with bad export cookie 13312606540767099800 [ 7590.730490] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7590.735622] LustreError: 228902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7591.584965] Lustre: server umount lustre-MDT0001 complete [ 7604.804205] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 7607.975727] Key type lgssc unregistered [ 7608.286824] LNet: 231024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7608.297879] LNetError: 231024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7608.317655] LNet: Removed LNI 192.168.202.131@tcp [ 7609.437313] Key type .llcrypt unregistered [ 7609.439840] Key type ._llcrypt unregistered [ 7635.948155] Key type ._llcrypt registered [ 7635.950167] Key type .llcrypt registered [ 7636.139797] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 7637.942030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7638.018267] alg: No test for adler32 (adler32-zlib) [ 7639.374423] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 7639.850026] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 7641.648456] Key type lgssc registered [ 7643.983989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7656.407878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7656.427728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7658.058028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7663.429366] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7668.198706] LustreError: 232435:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.226063] LustreError: 232435:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7673.320804] LustreError: 232436:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7673.579831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.986060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7679.104332] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7689.098589] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7697.307627] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7707.245629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7707.717639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7716.831759] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7726.813322] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7728.259298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7748.555374] Lustre: server umount lustre-OST0000 complete [ 7754.215956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7754.220268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7754.237685] Lustre: Skipped 1 previous similar message [ 7754.251504] Lustre: Skipped 1 previous similar message [ 7759.346443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7759.351752] Lustre: Skipped 1 previous similar message [ 7764.454092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7764.477658] Lustre: Skipped 1 previous similar message [ 7765.000858] Lustre: server umount lustre-MDT0000 complete [ 7765.472265] Lustre: 231746:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789057579/real 1789057579] req@00000000f66dc440 x1875962725702528/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789057595 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7765.503665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.541675] LustreError: 233439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7766.498441] Lustre: 231743:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789057580/real 1789057580] req@000000005b997974 x1875962725702912/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789057596 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7770.278602] LustreError: 232418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057600 with bad export cookie 14342408936332413805 [ 7770.285552] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7770.302868] LustreError: 232418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7770.865469] Lustre: server umount lustre-MDT0001 complete [ 7783.363700] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 7787.031402] Key type lgssc unregistered [ 7787.533199] LNet: 236127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7787.541920] LNetError: 236127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7787.573234] LNet: Removed LNI 192.168.202.131@tcp [ 7788.785079] Key type .llcrypt unregistered [ 7788.786368] Key type ._llcrypt unregistered [ 7812.980338] Key type ._llcrypt registered [ 7812.982898] Key type .llcrypt registered [ 7813.133336] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 7814.236560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7814.287607] alg: No test for adler32 (adler32-zlib) [ 7816.018395] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 7816.254253] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 7817.905191] Key type lgssc registered [ 7820.040342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8098.054222] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 8102.277356] Key type lgssc unregistered [ 8102.778288] LNet: 238145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8102.782981] LNetError: 238145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8102.805343] LNet: Removed LNI 192.168.202.131@tcp [ 8103.608545] Key type .llcrypt unregistered [ 8103.610470] Key type ._llcrypt unregistered [ 8132.876086] Key type ._llcrypt registered [ 8132.877971] Key type .llcrypt registered [ 8133.116382] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 8150.498376] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 8151.862872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8151.894031] alg: No test for adler32 (adler32-zlib) [ 8153.127484] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 8153.364688] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 8155.008328] Key type lgssc registered [ 8156.312069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8200.099053] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7894 sec ========== 12:33:48 (1789058028) [ 8201.953961] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:33:50 (1789058030) === [ 8205.963268] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:33:54 (1789058034) === [ 8252.311680] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 8256.327550] Key type lgssc unregistered [ 8256.786780] LNet: 243379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8256.794744] LNetError: 243379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8256.824666] LNet: Removed LNI 192.168.202.131@tcp [ 8258.177711] Key type .llcrypt unregistered [ 8258.179306] Key type ._llcrypt unregistered