[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 532596300 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002330] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007349] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010111] LSM: Security Framework initializing [ 0.011077] Yama: becoming mindful. [ 0.012025] SELinux: Initializing. [ 0.013049] *** VALIDATE selinux *** [ 0.021066] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026244] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028085] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029082] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031044] *** VALIDATE tmpfs *** [ 0.032465] *** VALIDATE proc *** [ 0.034058] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.037185] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038257] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040026] Spectre V2 : User space: Vulnerable [ 0.041000] Speculative Store Bypass: Vulnerable [ 0.042931] debug: unmapping init [mem 0xffffffff8f259000-0xffffffff8f260fff] [ 0.044157] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045586] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048008] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050008] ... max period: 00007fffffffffff [ 0.051010] ... fixed-purpose events: 3 [ 0.052007] ... event mask: 000000070000000f [ 0.053261] rcu: Hierarchical SRCU implementation. [ 0.055838] smp: Bringing up secondary CPUs ... [ 0.056480] x86: Booting SMP configuration: [ 0.057015] .... node #0, CPUs: #1 #2 #3 [ 0.060657] smp: Brought up 1 node, 4 CPUs [ 0.062011] smpboot: Max logical packages: 1 [ 0.063012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.106150] node 0 deferred pages initialised in 40ms [ 0.110362] devtmpfs: initialized [ 0.112413] x86/mm: Memory block size: 128MB [ 0.116748] gcov: version magic: 0x41383552 [ 0.120289] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.121066] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.123302] pinctrl core: initialized pinctrl subsystem [ 0.125138] [ 0.125537] ************************************************************* [ 0.127022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.129010] ** ** [ 0.131010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.133010] ** ** [ 0.135010] ** This means that this kernel is built to expose internal ** [ 0.137010] ** IOMMU data structures, which may compromise security on ** [ 0.139008] ** your system. ** [ 0.141010] ** ** [ 0.143011] ** If you see this message and you are not debugging the ** [ 0.145009] ** kernel, report this immediately to your vendor! ** [ 0.146008] ** ** [ 0.148009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150010] ************************************************************* [ 0.152729] NET: Registered protocol family 16 [ 0.154470] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.156052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.159047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.162030] cpuidle: using governor menu [ 0.163769] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.166426] PCI: Using configuration type 1 for base access [ 0.168169] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.177060] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.183041] cryptd: max_cpu_qlen set to 1000 [ 0.186648] ACPI: Added _OSI(Module Device) [ 0.188011] ACPI: Added _OSI(Processor Device) [ 0.190011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197507] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.204582] ACPI: Interpreter enabled [ 0.206048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207060] ACPI: Using IOAPIC for interrupt routing [ 0.209091] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.212831] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.227449] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.230076] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.232012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.235063] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245151] acpiphp: Slot [2] registered [ 0.248130] acpiphp: Slot [5] registered [ 0.251200] acpiphp: Slot [6] registered [ 0.253094] acpiphp: Slot [7] registered [ 0.255106] acpiphp: Slot [8] registered [ 0.256160] acpiphp: Slot [9] registered [ 0.258157] acpiphp: Slot [10] registered [ 0.259124] acpiphp: Slot [3] registered [ 0.261123] acpiphp: Slot [4] registered [ 0.262120] acpiphp: Slot [11] registered [ 0.263071] acpiphp: Slot [12] registered [ 0.265087] acpiphp: Slot [13] registered [ 0.266098] acpiphp: Slot [14] registered [ 0.268091] acpiphp: Slot [15] registered [ 0.270094] acpiphp: Slot [16] registered [ 0.272078] acpiphp: Slot [17] registered [ 0.273067] acpiphp: Slot [18] registered [ 0.274067] acpiphp: Slot [19] registered [ 0.275064] acpiphp: Slot [20] registered [ 0.276152] acpiphp: Slot [21] registered [ 0.278065] acpiphp: Slot [22] registered [ 0.279118] acpiphp: Slot [23] registered [ 0.280000] acpiphp: Slot [24] registered [ 0.280000] acpiphp: Slot [25] registered [ 0.282108] acpiphp: Slot [26] registered [ 0.283063] acpiphp: Slot [27] registered [ 0.284057] acpiphp: Slot [28] registered [ 0.287573] acpiphp: Slot [29] registered [ 0.289116] acpiphp: Slot [30] registered [ 0.290000] acpiphp: Slot [31] registered [ 0.292063] PCI host bridge to bus 0000:00 [ 0.293000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.298021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.301019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.304019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.307020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310164] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.313117] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.317313] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.334020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.339140] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.341028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.346033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.348038] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351623] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356022] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.359059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362860] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.370021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.385022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.392018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.399632] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.414024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.436022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.447000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.469000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.484016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.496478] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.509023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.525024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.540014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.549412] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.561022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.571022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.606021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.625662] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.634020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.643021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.665023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.677209] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.685014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.696020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.714014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.725859] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.728402] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.731535] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.733498] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.738093] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.744163] iommu: Default domain type: Passthrough [ 0.745000] SCSI subsystem initialized [ 0.745000] ACPI: bus type USB registered [ 0.746049] usbcore: registered new interface driver usbfs [ 0.747306] usbcore: registered new interface driver hub [ 0.750067] usbcore: registered new device driver usb [ 0.752182] pps_core: LinuxPPS API ver. 1 registered [ 0.754008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.758053] PTP clock support registered [ 0.761054] EDAC MC: Ver: 3.0.0 [ 0.762638] PCI: Using ACPI for IRQ routing [ 0.763000] NetLabel: Initializing [ 0.764017] NetLabel: domain hash size = 128 [ 0.766015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.768100] NetLabel: unlabeled traffic allowed by default [ 0.771383] vgaarb: loaded [ 0.772662] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.775014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.781789] clocksource: Switched to clocksource kvm-clock [ 0.890095] VFS: Disk quotas dquot_6.6.0 [ 0.891615] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.894755] *** VALIDATE ramfs *** [ 0.896522] *** VALIDATE hugetlbfs *** [ 0.898653] pnp: PnP ACPI init [ 0.902525] pnp: PnP ACPI: found 6 devices [ 0.923295] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.927098] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.929075] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.931023] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.933575] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.935865] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.939123] NET: Registered protocol family 2 [ 0.942092] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.948739] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.953321] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.961034] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.966208] TCP: Hash tables configured (established 65536 bind 65536) [ 0.969691] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.974877] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.978603] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.982479] NET: Registered protocol family 1 [ 0.985598] RPC: Registered named UNIX socket transport module. [ 0.987776] RPC: Registered udp transport module. [ 0.990060] RPC: Registered tcp transport module. [ 0.991938] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.994504] NET: Registered protocol family 44 [ 0.997218] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.999942] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.003858] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.007453] PCI: CLS 0 bytes, default 64 [ 1.010617] Unpacking initramfs... [ 2.505961] debug: unmapping init [mem 0xffff9dc67cc54000-0xffff9dc67ffbffff] [ 2.513227] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.515915] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.520673] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.035647] Initialise system trusted keyrings [ 3.037575] Key type blacklist registered [ 3.039678] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.049286] zbud: loaded [ 3.052479] *** VALIDATE nfs *** [ 3.053946] *** VALIDATE nfs4 *** [ 3.055570] pstore: using deflate compression [ 3.059849] Platform Keyring initialized [ 3.169806] NET: Registered protocol family 38 [ 3.171728] Key type asymmetric registered [ 3.173656] Asymmetric key parser 'x509' registered [ 3.175751] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.179289] io scheduler mq-deadline registered [ 3.181587] io scheduler kyber registered [ 3.183168] io scheduler bfq registered [ 3.185799] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.189888] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.193652] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.197363] ACPI: Power Button [PWRF] [ 3.204277] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.214969] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.237513] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.252755] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.274468] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.303132] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.331452] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.337906] Non-volatile memory driver v1.3 [ 3.340602] Linux agpgart interface v0.103 [ 3.378602] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.381416] vda: detected capacity change from 0 to 74698752 [ 3.397628] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.402219] vdb: detected capacity change from 0 to 1073741824 [ 3.425531] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.429505] vdc: detected capacity change from 0 to 2621440000 [ 3.446027] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.449482] vdd: detected capacity change from 0 to 2621440000 [ 3.468429] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.471801] vde: detected capacity change from 0 to 4294967296 [ 3.490678] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.493716] vdf: detected capacity change from 0 to 4294967296 [ 3.502943] libphy: Fixed MDIO Bus: probed [ 3.512708] usbcore: registered new interface driver usbserial_generic [ 3.516249] usbserial: USB Serial support registered for generic [ 3.518411] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.524243] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.526031] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.528767] mousedev: PS/2 mouse device common for all mice [ 3.532703] rtc_cmos 00:05: RTC can wake from S4 [ 3.535362] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.535732] rtc_cmos 00:05: registered as rtc0 [ 3.539770] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.541666] intel_pstate: CPU model not supported [ 3.543765] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.545762] hid: raw HID events driver (C) Jiri Kosina [ 3.551628] usbcore: registered new interface driver usbhid [ 3.553185] usbhid: USB HID core driver [ 3.554544] drop_monitor: Initializing network drop monitor service [ 3.555139] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.556436] Initializing XFRM netlink socket [ 3.561454] NET: Registered protocol family 10 [ 3.564344] Segment Routing with IPv6 [ 3.565289] NET: Registered protocol family 17 [ 3.566696] mpls_gso: MPLS GSO support [ 3.572761] RAS: Correctable Errors collector initialized. [ 3.574248] AVX version of gcm_enc/dec engaged. [ 3.575575] AES CTR mode by8 optimization enabled [ 3.644832] sched_clock: Marking stable (3644747808, 0)->(4524331385, -879583577) [ 3.648686] registered taskstats version 1 [ 3.651215] Loading compiled-in X.509 certificates [ 3.652950] zswap: loaded using pool lzo/zbud [ 3.680769] Key type big_key registered [ 3.697728] Key type encrypted registered [ 3.699448] ima: No TPM chip found, activating TPM-bypass! [ 3.701906] ima: Allocated hash algorithm: sha1 [ 3.703783] ima: No architecture policies found [ 3.705591] evm: Initialising EVM extended attributes: [ 3.707773] evm: security.selinux [ 3.709064] evm: security.ima [ 3.709995] evm: security.capability [ 3.712117] evm: HMAC attrs: 0x1 [ 3.714619] rtc_cmos 00:05: setting system clock to 2026-08-14 21:43:35 UTC (1786743815) [ 3.723525] debug: unmapping init [mem 0xffffffff90203000-0xffffffff903fffff] [ 3.726934] debug: unmapping init [mem 0xffffffff8ef82000-0xffffffff8f258fff] [ 3.736085] Write protecting the kernel read-only data: 28672k [ 3.739655] debug: unmapping init [mem 0xffffffff8d603000-0xffffffff8d7fffff] [ 3.742413] debug: unmapping init [mem 0xffffffff8df14000-0xffffffff8dffffff] [ 3.777206] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.784884] systemd[1]: Detected virtualization kvm. [ 3.786956] systemd[1]: Detected architecture x86-64. [ 3.789102] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.815038] systemd[1]: No hostname configured. [ 3.817034] systemd[1]: Set hostname to . [ 3.819363] random: systemd: uninitialized urandom read (16 bytes read) [ 3.822123] systemd[1]: Initializing machine ID from random generator. [ 3.870538] random: ln: uninitialized urandom read (6 bytes read) [ 3.980804] random: systemd: uninitialized urandom read (16 bytes read) [ 3.983883] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.988351] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.992787] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.713178] device-mapper: uevent: version 1.0.3 [ 4.718226] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.603790] virtio_net virtio0 ens2: renamed from eth0 [ 5.620744] random: fast init done [ 5.677753] scsi host0: ata_piix [ 5.708240] scsi host1: ata_piix [ 5.710137] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.713957] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.826969] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.218138] random: crng init done [ 10.220171] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.296953] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.695412] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.993634] SELinux: Disabled at runtime. [ 13.053139] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.063456] systemd[1]: Detected virtualization kvm. [ 13.065585] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.719713] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.725797] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.738149] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.745653] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.751448] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.760765] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.770881] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 13.843941] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 14.517852] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.892761] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.088464] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.250393] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.272798] EDAC sbridge: Ver: 1.1.2 [ 17.395347] Key type dns_resolver registered [ 17.766147] NFS: Registering the id_resolver key type [ 17.768433] Key type id_resolver registered [ 17.770622] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg636-server login: [ 68.835035] libcfs: loading out-of-tree module taints kernel. [ 68.942727] Key type ._llcrypt registered [ 68.945486] Key type .llcrypt registered [ 69.137024] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 85.019575] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 86.430976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.444871] alg: No test for adler32 (adler32-zlib) [ 87.848272] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 88.589486] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 90.359152] Key type lgssc registered [ 92.165255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 106.476122] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.016179] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 158.424928] hrtimer: interrupt took 7895669 ns [ 162.206431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.246433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.580988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.608808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.708435] Lustre: lustre-MDT0000: new disk, initializing [ 163.799410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.816141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.614703] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.675967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.810506] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 181.843358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 181.848572] Lustre: Skipped 1 previous similar message [ 181.959492] Lustre: lustre-MDT0001: new disk, initializing [ 182.026751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 182.068181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 182.083461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.319556] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.553587] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 202.123731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.607776] Lustre: lustre-OST0000: new disk, initializing [ 202.615046] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.624380] Lustre: 8457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.708352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.401205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.407714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.502871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.339472] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.176384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 227.390778] Lustre: lustre-OST0001: new disk, initializing [ 227.395309] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 227.401514] Lustre: 9529:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.499731] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 234.001424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 234.012577] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 234.116904] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 234.173200] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.427613] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.118810] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 257.926800] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing check_logdir /tmp/testlogs/ [ 263.348540] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing yml_node [ 266.901346] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 269.347883] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 271.546377] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 272.946930] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 17:48:02 EDT 2026 [ 290.289761] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 292.170901] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 310.239655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 310.255764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.278075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.508557] Lustre: server umount lustre-MDT0000 complete [ 317.148128] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744128 with bad export cookie 12987391345370227434 [ 317.169508] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 317.169750] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 321.002713] LustreError: 6524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.008843] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.009472] LustreError: 6524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 321.011457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.011464] Lustre: Skipped 3 previous similar messages [ 321.045411] Lustre: Skipped 4 previous similar messages [ 323.039671] LustreError: 6525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.042853] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.071839] LustreError: 6525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 323.101399] Lustre: Skipped 2 previous similar messages [ 323.542537] Lustre: server umount lustre-MDT0001 complete [ 328.150033] Lustre: server umount lustre-OST0000 complete [ 332.124769] Lustre: server umount lustre-OST0001 complete [ 338.632586] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 348.584150] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 387.525962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.807636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 387.861553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 387.935498] Lustre: lustre-MDT0000: new disk, initializing [ 388.016875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.032660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 392.807873] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 403.209944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.309641] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 403.379070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 403.385696] Lustre: Skipped 1 previous similar message [ 403.514264] Lustre: lustre-MDT0001: new disk, initializing [ 403.672512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 403.731027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 403.748055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 408.799078] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 418.761862] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 425.016215] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 432.708489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 433.216938] Lustre: lustre-OST0000: new disk, initializing [ 433.224029] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 433.232514] Lustre: 17052:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 433.413078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.652312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 434.674517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 434.817993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 440.046558] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 450.627409] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 455.537461] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 455.735440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 461.111343] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 461.320475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 465.382528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.392481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.401591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.420067] Lustre: Skipped 1 previous similar message [ 469.622387] Lustre: server umount lustre-OST0000 complete [ 475.618376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 475.624625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.633515] LustreError: Skipped 1 previous similar message [ 475.635381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 475.635388] Lustre: Skipped 1 previous similar message [ 475.663826] Lustre: Skipped 1 previous similar message [ 480.032183] Lustre: server umount lustre-MDT0000 complete [ 480.736991] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.747231] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 483.943782] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744295 with bad export cookie 12987391345370228967 [ 483.952624] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 483.957520] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 484.323203] Lustre: server umount lustre-MDT0001 complete [ 488.203713] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:51:37 (1786744297) [ 495.757723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.350567] LustreError: 19321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.437928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.157786] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 501.741695] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.688166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 514.949618] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 523.234449] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 529.282696] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 536.468763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 544.844732] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 554.895658] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 564.718290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.733321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 564.743609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 564.747471] Lustre: Skipped 1 previous similar message [ 564.783271] Lustre: Skipped 2 previous similar messages [ 569.330545] Lustre: server umount lustre-OST0000 complete [ 574.945191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.035390] Lustre: server umount lustre-MDT0000 complete [ 580.129765] LustreError: 19321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.143909] LustreError: 19321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 584.436641] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744396 with bad export cookie 12987391345370229849 [ 584.438395] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 584.456916] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 584.880303] Lustre: server umount lustre-MDT0001 complete [ 598.138464] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 601.774642] Key type lgssc unregistered [ 602.175606] LNet: 22858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.184427] LNetError: 22858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.214645] LNet: Removed LNI 192.168.206.136@tcp [ 603.435337] Key type .llcrypt unregistered [ 603.442462] Key type ._llcrypt unregistered [ 631.427329] Key type ._llcrypt registered [ 631.431906] Key type .llcrypt registered [ 631.592310] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 632.946790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.027032] alg: No test for adler32 (adler32-zlib) [ 634.403671] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 634.817276] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 636.639101] Key type lgssc registered [ 638.038646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 646.997322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 647.020453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.410469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 653.452110] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 658.918068] LustreError: 24278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.947219] LustreError: 24278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 662.443297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.899373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 667.808785] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 676.996550] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 683.189823] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 689.820393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 690.058466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 694.199251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 696.422387] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 704.666378] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 722.920908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.929075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 722.935643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 722.945785] Lustre: Skipped 1 previous similar message [ 728.042768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 728.056671] Lustre: Skipped 1 previous similar message [ 728.299721] Lustre: server umount lustre-OST0000 complete [ 733.153929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.157219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 733.169813] Lustre: Skipped 2 previous similar messages [ 733.185709] Lustre: Skipped 1 previous similar message [ 738.274245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 738.286581] Lustre: Skipped 1 previous similar message [ 738.835746] Lustre: server umount lustre-MDT0000 complete [ 742.932329] LustreError: 27303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744554 with bad export cookie 13050792403533508489 [ 742.936237] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.939325] LustreError: 27303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 743.301202] Lustre: server umount lustre-MDT0001 complete [ 755.794355] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 758.998938] Key type lgssc unregistered [ 759.354777] LNet: 28259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.367372] LNetError: 28259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.407533] LNet: Removed LNI 192.168.206.136@tcp [ 760.511635] Key type .llcrypt unregistered [ 760.513425] Key type ._llcrypt unregistered [ 785.666507] Key type ._llcrypt registered [ 785.670492] Key type .llcrypt registered [ 785.842314] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 786.701320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 786.766700] alg: No test for adler32 (adler32-zlib) [ 787.869691] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 788.225171] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 789.975203] Key type lgssc registered [ 791.611939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 800.224751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 800.248801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 801.706397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 806.603041] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 812.004220] LustreError: 29672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 812.022187] LustreError: 29672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 815.429460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.990844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 820.807650] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 828.555993] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 834.477693] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 849.107483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 849.588102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 856.475812] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 865.019895] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 872.116069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 882.660749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 882.663159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 882.690848] Lustre: Skipped 1 previous similar message [ 887.778405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 887.793075] Lustre: Skipped 2 previous similar messages [ 888.083758] Lustre: server umount lustre-OST0000 complete [ 892.899378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.899969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 892.926156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 892.934117] Lustre: Skipped 1 previous similar message [ 898.019376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 898.026644] Lustre: Skipped 1 previous similar message [ 903.137929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 903.141305] Lustre: Skipped 1 previous similar message [ 906.207153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 906.462338] Lustre: server umount lustre-MDT0000 complete [ 908.261650] LustreError: 29672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.286259] LustreError: 29672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 910.379863] LustreError: 29652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744722 with bad export cookie 5590981152688516642 [ 910.396600] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 910.851798] Lustre: server umount lustre-MDT0001 complete [ 922.069521] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 924.707902] Key type lgssc unregistered [ 925.074161] LNet: 33578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.086670] LNetError: 33578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.123353] LNet: Removed LNI 192.168.206.136@tcp [ 926.951590] Key type .llcrypt unregistered [ 926.953634] Key type ._llcrypt unregistered [ 947.822917] Key type ._llcrypt registered [ 947.829685] Key type .llcrypt registered [ 947.926684] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 948.884878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.917249] alg: No test for adler32 (adler32-zlib) [ 950.062952] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 950.324831] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 951.983134] Key type lgssc registered [ 952.898494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 961.030301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 961.050206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.388193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 967.348921] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 972.768469] LustreError: 34973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.805420] LustreError: 34973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 976.294574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.733951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 980.786506] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 989.884054] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 996.089987] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1002.973864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1003.465366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.548592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1010.762142] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1019.819481] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1032.172213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1032.180510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.194935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1032.237168] Lustre: Skipped 1 previous similar message [ 1035.909360] Lustre: server umount lustre-OST0000 complete [ 1042.407655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.418158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1042.428987] Lustre: Skipped 1 previous similar message [ 1042.436513] Lustre: Skipped 1 previous similar message [ 1046.576911] Lustre: server umount lustre-MDT0000 complete [ 1047.520839] LustreError: 35702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1047.543655] LustreError: 35702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1050.795051] LustreError: 34953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744862 with bad export cookie 12247537111345833659 [ 1050.799470] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1050.813782] LustreError: 34953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1051.146685] Lustre: server umount lustre-MDT0001 complete [ 1064.044303] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1067.721414] Key type lgssc unregistered [ 1068.125952] LNet: 38523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.145057] LNetError: 38523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.177731] LNet: Removed LNI 192.168.206.136@tcp [ 1069.371284] Key type .llcrypt unregistered [ 1069.373403] Key type ._llcrypt unregistered [ 1093.240909] Key type ._llcrypt registered [ 1093.244320] Key type .llcrypt registered [ 1093.369707] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1094.118139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1094.142528] alg: No test for adler32 (adler32-zlib) [ 1095.368669] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1095.646123] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1097.375181] Key type lgssc registered [ 1098.833666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1107.897759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1107.930866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.433184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.829879] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1119.715534] LustreError: 39936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.741888] LustreError: 39936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1124.837283] LustreError: 39937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1124.873846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1125.259235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1129.488723] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1137.252684] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1142.888715] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1150.264147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1150.616519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.764597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1156.702827] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1164.106408] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1175.522419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1175.529474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.545719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1175.569816] Lustre: Skipped 1 previous similar message [ 1176.891943] Lustre: server umount lustre-OST0000 complete [ 1199.072897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.075412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.082479] Lustre: Skipped 2 previous similar messages [ 1199.092932] Lustre: Skipped 1 previous similar message [ 1200.100164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.108963] Lustre: Skipped 1 previous similar message [ 1203.574301] Lustre: server umount lustre-MDT0000 complete [ 1205.217637] LustreError: 39936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.240571] LustreError: 39936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1207.385844] LustreError: 39917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745019 with bad export cookie 11008023504345117029 [ 1207.394178] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1213.892353] Lustre: server umount lustre-MDT0001 complete [ 1225.372980] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1228.417477] Key type lgssc unregistered [ 1228.698637] LNet: 43645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.718253] LNetError: 43645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.740319] LNet: Removed LNI 192.168.206.136@tcp [ 1230.738882] Key type .llcrypt unregistered [ 1230.743551] Key type ._llcrypt unregistered [ 1254.028175] Key type ._llcrypt registered [ 1254.031439] Key type .llcrypt registered [ 1254.205929] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1255.460574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1255.565850] alg: No test for adler32 (adler32-zlib) [ 1256.676194] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1256.993757] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1258.687221] Key type lgssc registered [ 1260.026778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1267.449928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1267.463788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.934416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.003543] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1278.436686] LustreError: 45061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.444699] LustreError: 45061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1281.989651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.271879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1286.282584] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1293.778162] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1299.139185] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1306.019953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1306.450238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1312.840683] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1313.600116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1321.030416] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1328.179098] Lustre: server umount lustre-MDT0000 complete [ 1328.608099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1328.613683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.632144] LustreError: 45061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1329.124607] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.149018] Lustre: Skipped 1 previous similar message [ 1332.463240] LustreError: 45041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745144 with bad export cookie 11056232635273230662 [ 1332.485607] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1333.862448] LustreError: 45056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1333.881490] LustreError: 45056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1333.891646] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.36@tcp (stopping) [ 1334.249282] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1337.251067] LustreError: 45057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.251710] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1337.272766] LustreError: 45057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1337.300679] Lustre: Skipped 1 previous similar message [ 1338.801963] Lustre: server umount lustre-MDT0001 complete [ 1359.985261] Lustre: server umount lustre-OST0000 complete [ 1379.064848] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1382.552744] Key type lgssc unregistered [ 1382.825057] LNet: 48923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.833094] LNetError: 48923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.852203] LNet: Removed LNI 192.168.206.136@tcp [ 1384.978536] Key type .llcrypt unregistered [ 1384.980622] Key type ._llcrypt unregistered [ 1408.163388] Key type ._llcrypt registered [ 1408.166357] Key type .llcrypt registered [ 1408.289823] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1409.547939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1409.583087] alg: No test for adler32 (adler32-zlib) [ 1410.767951] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1411.008204] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1412.680492] Key type lgssc registered [ 1414.275865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1422.908723] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1422.948152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1438.559392] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1443.679273] LustreError: 50316:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.136@tcp: failed processing log, type 4: rc = -110 [ 1470.306434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.539873] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1483.888076] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1649.091596] Lustre: server umount lustre-OST0000 complete [ 1669.534841] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1672.972789] Key type lgssc unregistered [ 1673.506109] LNet: 51998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1673.519560] LNetError: 51998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1673.538585] LNet: Removed LNI 192.168.206.136@tcp [ 1674.717513] Key type .llcrypt unregistered [ 1674.724841] Key type ._llcrypt unregistered [ 1697.668779] Key type ._llcrypt registered [ 1697.673393] Key type .llcrypt registered [ 1697.932973] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1699.345748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1699.370565] alg: No test for adler32 (adler32-zlib) [ 1700.478486] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1700.764285] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1702.463471] Key type lgssc registered [ 1703.699156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1712.569603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1712.585226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1714.107992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1719.019147] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1723.362888] LustreError: 53391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1723.397150] LustreError: 53391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1728.483176] LustreError: 53392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1728.534136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.019372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1734.705359] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1742.956484] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1749.687327] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1757.892546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1758.489657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.558865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1765.837447] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1773.527222] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1780.193056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.200550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.207269] LustreError: Skipped 1 previous similar message [ 1780.225276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.005178] Lustre: server umount lustre-OST0000 complete [ 1785.316537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1785.319441] LustreError: 52700:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000debcd9a6 x1873538462018688/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1785.323842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.323850] Lustre: Skipped 1 previous similar message [ 1785.359706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.364377] Lustre: Skipped 2 previous similar messages [ 1790.445973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1790.451299] Lustre: Skipped 1 previous similar message [ 1791.675623] Lustre: server umount lustre-MDT0000 complete [ 1794.930579] LustreError: 53372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745606 with bad export cookie 7091509314080583857 [ 1794.943730] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1795.225139] Lustre: server umount lustre-MDT0001 complete [ 1805.460167] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1808.159714] Key type lgssc unregistered [ 1808.519218] LNet: 56929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.530101] LNetError: 56929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.564501] LNet: Removed LNI 192.168.206.136@tcp [ 1809.312206] Key type .llcrypt unregistered [ 1809.315376] Key type ._llcrypt unregistered [ 1838.555350] Key type ._llcrypt registered [ 1838.556832] Key type .llcrypt registered [ 1838.661601] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1840.198139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1840.261881] alg: No test for adler32 (adler32-zlib) [ 1841.456133] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1841.691405] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1843.416210] Key type lgssc registered [ 1844.519375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1853.424875] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1853.467962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1869.087625] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1874.207317] LustreError: 58314:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.136@tcp: failed processing log, type 4: rc = -110 [ 1900.895850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.483263] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1915.809864] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1923.895488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1924.609389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.692341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1929.705366] LustreError: 58339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1929.736016] LustreError: 58339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1930.892935] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1934.831582] LustreError: 58340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.857765] LustreError: 59809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.899890] LustreError: 59809:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1940.987132] LustreError: 59350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1941.005922] LustreError: 59350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1942.723867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.296406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1948.286516] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1956.242696] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1962.536901] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1965.536529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1965.550179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.571158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1969.152629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1969.155390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1969.169753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.192462] Lustre: Skipped 1 previous similar message [ 1974.242625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1974.250192] Lustre: Skipped 1 previous similar message [ 1979.359148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1979.370772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1979.393227] Lustre: Skipped 1 previous similar message [ 1979.574752] Lustre: server umount lustre-OST0000 complete [ 2004.961710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.968349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2004.975122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2005.004597] Lustre: Skipped 1 previous similar message [ 2007.405706] Lustre: server umount lustre-MDT0000 complete [ 2010.081064] LustreError: 61056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.093107] LustreError: 61056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2011.047081] LustreError: 58322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745822 with bad export cookie 13057075472928669703 [ 2011.050427] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2011.067983] LustreError: 58322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2017.557578] Lustre: server umount lustre-MDT0001 complete [ 2029.123891] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2032.081261] Key type lgssc unregistered [ 2032.345679] LNet: 62063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.361284] LNetError: 62063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.386077] LNet: Removed LNI 192.168.206.136@tcp [ 2034.336840] Key type .llcrypt unregistered [ 2034.340810] Key type ._llcrypt unregistered [ 2059.106200] Key type ._llcrypt registered [ 2059.112313] Key type .llcrypt registered [ 2059.278320] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2060.828091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2060.888088] alg: No test for adler32 (adler32-zlib) [ 2062.514080] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2063.042817] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2064.919987] Key type lgssc registered [ 2066.661381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2076.100720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2076.136043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.495711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.229892] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2087.906159] LustreError: 63478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.921743] LustreError: 63478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2091.114299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2091.524064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2096.605046] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2104.806775] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2111.024175] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2117.209175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2117.454725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.601759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2122.974124] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2132.387537] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2152.333485] Lustre: server umount lustre-OST0000 complete [ 2157.536709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.539249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2157.550211] Lustre: Skipped 1 previous similar message [ 2157.559812] Lustre: Skipped 1 previous similar message [ 2162.657824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2162.666429] Lustre: Skipped 1 previous similar message [ 2166.836311] Lustre: server umount lustre-MDT0000 complete [ 2167.784529] LustreError: 63477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.805237] LustreError: 63477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2168.804154] Lustre: 62788:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786745964/real 1786745964] req@000000006658d7ac x1873538841603328/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786745980 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2168.835943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.853890] LustreError: 63496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.868423] LustreError: 63496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2170.387936] LustreError: 63459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745982 with bad export cookie 8873471961402739162 [ 2170.397747] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2170.744647] Lustre: server umount lustre-MDT0001 complete [ 2180.710741] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2184.404779] Key type lgssc unregistered [ 2184.759892] LNet: 67041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2184.768770] LNetError: 67041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2184.792293] LNet: Removed LNI 192.168.206.136@tcp [ 2185.937589] Key type .llcrypt unregistered [ 2185.940587] Key type ._llcrypt unregistered [ 2218.071680] Key type ._llcrypt registered [ 2218.079219] Key type .llcrypt registered [ 2218.218530] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2219.411574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2219.542815] alg: No test for adler32 (adler32-zlib) [ 2220.693293] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2220.986482] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2222.671376] Key type lgssc registered [ 2223.784771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2231.467564] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 18:20:41 (1786746041) [ 2238.403588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2238.437615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.023126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.821214] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2250.220504] LustreError: 68746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.257557] LustreError: 68746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2254.044169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.458726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2259.182147] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2266.710824] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2272.747530] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2279.666444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2280.108599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.234022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2285.568176] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2293.070784] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2302.471757] Lustre: Failing over lustre-MDT0000 [ 2303.028940] Lustre: server umount lustre-MDT0000 complete [ 2304.996813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.001659] LustreError: 68741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.022566] Lustre: Skipped 1 previous similar message [ 2305.059671] LustreError: 68741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2310.123751] LustreError: 68742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.152298] LustreError: 68742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2314.583051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2314.875418] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.097797] Lustre: *** cfs_fail_loc=135, val=0*** [ 2315.222610] LustreError: 71834:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2315.229600] LustreError: 71834:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.136@tcp: cfg command failed: rc = -2 [ 2315.236319] LustreError: 68747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.251841] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2315.251841] [ 2315.254059] LustreError: MGC192.168.206.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2315.254083] LustreError: 71822:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2315.254187] LustreError: 71822:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2315.254668] LustreError: 71822:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2315.254682] LustreError: 71822:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2315.260249] LustreError: 68727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746127 with bad export cookie 7488510895263910187 [ 2315.292351] Lustre: server umount lustre-MDT0000 complete [ 2315.309788] LustreError: 68747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2315.353390] LustreError: 71822:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2320.362071] LustreError: 69474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.379319] LustreError: 69474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2322.586786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.512092] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x67ec88fbc0d909be [ 2325.535382] Lustre: MGC192.168.206.136@tcp: Connection restored to 0@lo (at 0@lo) [ 2325.598373] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.36@tcp (not set up) [ 2325.750028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.799439] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2326.625243] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2329.982464] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2331.119903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2331.149721] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2331.188336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2340.833250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2340.838744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.846782] Lustre: Skipped 1 previous similar message [ 2340.850321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2343.893421] Lustre: server umount lustre-OST0000 complete [ 2351.588979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.589526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2351.605974] Lustre: Skipped 1 previous similar message [ 2351.607271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.629680] Lustre: Skipped 3 previous similar messages [ 2356.715519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2356.728485] Lustre: Skipped 1 previous similar message [ 2361.825220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.839866] Lustre: Skipped 1 previous similar message [ 2362.335520] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2362.510734] Lustre: server umount lustre-MDT0000 complete [ 2366.945988] LustreError: 68743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.969667] LustreError: 68743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2367.107117] LustreError: 68727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746178 with bad export cookie 7488510895263910334 [ 2367.115097] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2367.115128] LustreError: 68727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2367.120324] LustreError: Skipped 1 previous similar message [ 2367.614509] Lustre: server umount lustre-MDT0001 complete [ 2379.465547] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2382.957845] Key type lgssc unregistered [ 2383.305552] LNet: 73620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2383.314295] LNetError: 73620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2383.329356] LNet: Removed LNI 192.168.206.136@tcp [ 2384.161109] Key type .llcrypt unregistered [ 2384.165099] Key type ._llcrypt unregistered [ 2407.172072] Key type ._llcrypt registered [ 2407.174068] Key type .llcrypt registered [ 2407.282489] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2408.139811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2408.186856] alg: No test for adler32 (adler32-zlib) [ 2409.313481] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2409.596820] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2411.311297] Key type lgssc registered [ 2412.272298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2420.896419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2420.915597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.486681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.203768] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2432.995914] LustreError: 75034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.015937] LustreError: 75034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2435.469972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.886477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2439.982540] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2447.354593] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2453.395105] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2460.277816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2460.715673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.032904] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2467.841388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2475.964606] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2481.432696] Lustre: Failing over lustre-MDT0000 [ 2481.844354] Lustre: server umount lustre-MDT0000 complete [ 2482.145731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2482.177323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.206364] LustreError: 75034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.172221] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.183357] Lustre: Skipped 1 previous similar message [ 2487.909622] LustreError: 75028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.924247] LustreError: 75028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2491.560900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.686447] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2491.915703] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2492.081790] LustreError: 78123:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2492.092729] LustreError: 78123:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.136@tcp: cfg command failed: rc = -9 [ 2492.100708] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2492.100708] [ 2492.112919] LustreError: MGC192.168.206.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2492.125629] LustreError: 78111:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2492.142885] LustreError: 78111:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2492.152671] LustreError: 78111:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2492.155351] LustreError: 78111:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2492.181565] Lustre: server umount lustre-MDT0000 complete [ 2492.187620] LustreError: 78111:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2493.021462] LustreError: 76088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.059922] LustreError: 76088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2498.165859] LustreError: 76088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.195526] LustreError: 76088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2499.220336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2499.424391] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2499.790366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.832840] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2503.262143] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2504.578743] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2505.204118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2505.239047] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2505.267603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2511.842297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2511.857582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.875809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2515.431745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2515.434111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.874690] Lustre: server umount lustre-OST0000 complete [ 2525.665755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.667792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2525.687452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2525.695778] Lustre: Skipped 2 previous similar messages [ 2527.749665] Lustre: server umount lustre-MDT0000 complete [ 2530.786084] LustreError: 75028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.800568] LustreError: 75028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2531.901154] LustreError: 75014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746343 with bad export cookie 1988566170392962842 [ 2531.917499] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2532.262665] Lustre: server umount lustre-MDT0001 complete [ 2543.085371] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2545.801989] Key type lgssc unregistered [ 2546.101682] LNet: 79906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2546.113917] LNetError: 79906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2546.137479] LNet: Removed LNI 192.168.206.136@tcp [ 2547.037462] Key type .llcrypt unregistered [ 2547.042883] Key type ._llcrypt unregistered [ 2560.605352] Key type ._llcrypt registered [ 2560.608678] Key type .llcrypt registered [ 2560.727819] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2561.036291] Key type .llcrypt unregistered [ 2561.042299] Key type ._llcrypt unregistered [ 2586.287402] Key type ._llcrypt registered [ 2586.289478] Key type .llcrypt registered [ 2586.374950] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2587.656140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2587.739546] alg: No test for adler32 (adler32-zlib) [ 2589.077265] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2589.421642] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2591.129181] Key type lgssc registered [ 2592.795808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2601.676945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2601.691402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.225206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2607.968567] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2612.718401] LustreError: 81812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.738275] LustreError: 81812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2616.782613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.186841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2621.811678] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2628.660984] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2634.992911] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2642.904086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2643.453513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2647.563750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2650.584953] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2658.413769] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2672.111766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2672.115933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.126788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2672.127621] Lustre: Skipped 1 previous similar message [ 2672.135478] Lustre: Skipped 1 previous similar message [ 2677.203194] Lustre: server umount lustre-OST0000 complete [ 2682.348057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.363796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2682.371389] Lustre: Skipped 1 previous similar message [ 2687.429668] Lustre: server umount lustre-MDT0000 complete [ 2687.457876] LustreError: 81810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.487637] LustreError: 81810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2690.885579] LustreError: 84304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746502 with bad export cookie 13332623480878598297 [ 2690.894913] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2690.902652] LustreError: 84304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2691.325909] Lustre: server umount lustre-MDT0001 complete [ 2701.670382] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2704.115535] Key type lgssc unregistered [ 2704.389507] LNet: 85368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2704.397380] LNetError: 85368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2704.423695] LNet: Removed LNI 192.168.206.136@tcp [ 2705.397112] Key type .llcrypt unregistered [ 2705.403330] Key type ._llcrypt unregistered [ 2728.323326] Key type ._llcrypt registered [ 2728.325190] Key type .llcrypt registered [ 2728.433570] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2729.770141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2729.940705] alg: No test for adler32 (adler32-zlib) [ 2731.042080] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2731.292106] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2732.999364] Key type lgssc registered [ 2734.119291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2741.267193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2741.282076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.634930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2747.793322] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2752.994217] LustreError: 86779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.005323] LustreError: 86779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2756.022284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2756.588897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2761.783880] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2769.483321] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2774.811557] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2781.669363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2782.224600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2788.230473] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2788.344303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2795.900587] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2802.659053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2802.666603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.680204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2803.681663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2803.693547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2803.704465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2803.715963] Lustre: Skipped 1 previous similar message [ 2804.842748] Lustre: server umount lustre-OST0000 complete [ 2808.800126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.815713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2808.818308] Lustre: Skipped 1 previous similar message [ 2808.834456] Lustre: Skipped 1 previous similar message [ 2813.927108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2813.932013] Lustre: Skipped 1 previous similar message [ 2814.483983] Lustre: server umount lustre-MDT0000 complete [ 2818.308990] LustreError: 86761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746630 with bad export cookie 17828711650653713004 [ 2818.314658] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2818.318543] LustreError: 86761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2818.952888] Lustre: server umount lustre-MDT0001 complete [ 2830.436823] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2833.186252] Key type lgssc unregistered [ 2833.583721] LNet: 90317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2833.601690] LNetError: 90317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2833.618366] LNet: Removed LNI 192.168.206.136@tcp [ 2834.416514] Key type .llcrypt unregistered [ 2834.418860] Key type ._llcrypt unregistered [ 2857.055844] Key type ._llcrypt registered [ 2857.057961] Key type .llcrypt registered [ 2857.179601] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2857.752476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2857.779971] alg: No test for adler32 (adler32-zlib) [ 2858.753695] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2859.021665] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2860.735263] Key type lgssc registered [ 2861.729593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2868.579189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2868.601914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.966930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.020106] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2879.473256] LustreError: 91729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.487081] LustreError: 91729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2882.387075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.726785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2886.105773] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2892.262794] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2897.975854] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2904.522136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2904.848978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.774285] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2915.187934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2917.757995] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2928.609653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2928.616640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.625421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2930.149717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2930.158329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.174147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2933.515961] Lustre: server umount lustre-OST0000 complete [ 2938.850797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2938.855301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.869047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2938.872427] Lustre: Skipped 1 previous similar message [ 2940.897161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.900257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2940.944310] Lustre: Skipped 1 previous similar message [ 2942.772840] Lustre: server umount lustre-MDT0000 complete [ 2946.031051] LustreError: 92469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.047484] LustreError: 92469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2946.231967] LustreError: 91711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746758 with bad export cookie 6801273590899418787 [ 2946.234837] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2946.239846] LustreError: 91711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2946.493519] Lustre: server umount lustre-MDT0001 complete [ 2955.643736] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2958.217312] Key type lgssc unregistered [ 2958.465993] LNet: 95263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2958.471908] LNetError: 95263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2958.495615] LNet: Removed LNI 192.168.206.136@tcp [ 2959.191886] Key type .llcrypt unregistered [ 2959.194234] Key type ._llcrypt unregistered [ 2977.995268] Key type ._llcrypt registered [ 2977.999362] Key type .llcrypt registered [ 2978.104132] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2979.158661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2979.193717] alg: No test for adler32 (adler32-zlib) [ 2980.276700] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2980.500734] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2982.167535] Key type lgssc registered [ 2983.248909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2990.184789] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2990.202543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3005.793754] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3010.911430] LustreError: 96631:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.136@tcp: failed processing log, type 4: rc = -110 [ 3037.537384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3043.157569] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3049.288523] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3054.180626] Lustre: server umount lustre-OST0000 complete [ 3061.324670] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:34:31 (1786746871) [ 3067.275032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3067.681066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.693549] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3073.002815] LustreError: 98268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3073.017085] LustreError: 98268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3078.119506] LustreError: 98269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3079.688426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.940290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3084.333302] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3091.374672] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3097.981095] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3105.187152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3105.515149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.226801] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3112.693157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3120.631238] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3138.529125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3138.538574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.547339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.578129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.584203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3140.599129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.264318] Lustre: server umount lustre-OST0000 complete [ 3145.699546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.703353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3145.715107] Lustre: Skipped 1 previous similar message [ 3145.718346] Lustre: Skipped 1 previous similar message [ 3150.817971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3150.825469] Lustre: Skipped 2 previous similar messages [ 3151.466778] Lustre: server umount lustre-MDT0000 complete [ 3154.975266] LustreError: 98250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746966 with bad export cookie 1431198944860397438 [ 3154.989287] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.479326] Lustre: server umount lustre-MDT0001 complete [ 3165.752951] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 3168.340527] Key type lgssc unregistered [ 3168.605951] LNet: 101965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3168.617197] LNetError: 101965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3168.646710] LNet: Removed LNI 192.168.206.136@tcp [ 3169.513512] Key type .llcrypt unregistered [ 3169.515289] Key type ._llcrypt unregistered [ 3206.158176] Key type ._llcrypt registered [ 3206.160216] Key type .llcrypt registered [ 3206.323706] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3207.312123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3207.375896] alg: No test for adler32 (adler32-zlib) [ 3208.532823] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 3208.912717] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 3210.591222] Key type lgssc registered [ 3211.977693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3219.374638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3219.394934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.779570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.208899] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3231.211455] LustreError: 104167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3231.233452] LustreError: 104167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3234.299092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3234.626933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3239.644972] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3247.149833] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3253.804120] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3260.608484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3260.852334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.813686] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3267.004236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3273.281828] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3287.016434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3287.020657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.029830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3289.569324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3289.579870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.584174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3289.595292] Lustre: Skipped 1 previous similar message [ 3292.547649] Lustre: server umount lustre-OST0000 complete [ 3296.224435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3296.231703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3296.244259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.821695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.827696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.536397] Lustre: server umount lustre-MDT0000 complete [ 3304.939139] LustreError: 104167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.955289] LustreError: 104167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3305.855483] LustreError: 106009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747117 with bad export cookie 13765895090476843512 [ 3305.858576] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3305.866629] LustreError: 106009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3306.131077] Lustre: server umount lustre-MDT0001 complete [ 3317.145968] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 3319.987588] Key type lgssc unregistered [ 3320.305208] LNet: 107724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3320.318176] LNetError: 107724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3320.336385] LNet: Removed LNI 192.168.206.136@tcp [ 3321.286791] Key type .llcrypt unregistered [ 3321.290072] Key type ._llcrypt unregistered [ 3338.967922] Key type ._llcrypt registered [ 3338.970863] Key type .llcrypt registered [ 3339.078323] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3339.898471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3339.947397] alg: No test for adler32 (adler32-zlib) [ 3341.075876] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 3341.360141] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 3343.033825] Key type lgssc registered [ 3344.007577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3351.036243] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3351.077873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3366.687686] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3371.808111] LustreError: 108966:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.136@tcp: failed processing log, type 4: rc = -110 [ 3398.495834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.025865] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3410.763963] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3417.386161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3417.618097] LustreError: MGC192.168.206.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3417.627577] LustreError: 109989:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3417.632569] LustreError: 109989:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3417.645103] LustreError: 109989:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3417.768342] Lustre: server umount lustre-MDT0000 complete [ 3417.771325] LustreError: 109989:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3439.071517] Lustre: 108303:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786747234/real 1786747234] req@00000000f1d806f9 x1873540182707584/t0(0) o400->MGC192.168.206.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786747250 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3439.109975] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3439.121503] LustreError: 108978:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3459.645330] Lustre: server umount lustre-OST0000 complete [ 3469.077448] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 3476.329555] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3507.666634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.853112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3507.881417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3507.955828] Lustre: lustre-MDT0000: new disk, initializing [ 3508.028716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.043723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3511.608413] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3520.583106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.733763] Lustre: 112783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3520.783825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3520.787737] Lustre: Skipped 1 previous similar message [ 3520.861684] Lustre: lustre-MDT0001: new disk, initializing [ 3520.966700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3520.999413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3521.010220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3524.624993] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3533.223991] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3538.093379] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3544.504412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3544.780728] Lustre: lustre-OST0000: new disk, initializing [ 3544.784337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3544.789724] Lustre: 114758:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3544.904743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.261126] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3554.850184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3554.856771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3554.922152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3558.105274] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3562.180241] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3562.402679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3566.975253] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3567.201689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3570.153883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3570.156982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3570.157197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3570.160777] Lustre: Skipped 1 previous similar message [ 3570.186416] LustreError: Skipped 1 previous similar message [ 3575.008436] Lustre: server umount lustre-OST0000 complete [ 3580.387604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.407032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3580.412849] Lustre: Skipped 2 previous similar messages [ 3584.429796] Lustre: server umount lustre-MDT0000 complete [ 3587.656344] LustreError: 112775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747399 with bad export cookie 10176061776268992844 [ 3587.664561] LustreError: 112775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3587.664746] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3587.944642] Lustre: server umount lustre-MDT0001 complete [ 3595.939823] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:43:25 (1786747405) [ 3599.183484] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3613.732357] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 3623.214758] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3652.784892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3652.956390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3652.976141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3653.028836] Lustre: lustre-MDT0000: new disk, initializing [ 3653.097456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.110319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3657.357798] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3666.849932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.948870] Lustre: 119656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3667.000692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3667.003935] Lustre: Skipped 1 previous similar message [ 3667.073384] Lustre: lustre-MDT0001: new disk, initializing [ 3667.138096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3667.162650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3667.172927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3670.680826] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3678.376243] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3682.769309] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3687.903607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3688.143560] Lustre: lustre-OST0000: new disk, initializing [ 3688.146854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3688.152316] Lustre: 121621:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3688.200533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.643545] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3695.647604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3695.663527] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3695.787795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3699.973826] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3703.741960] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3703.954699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3708.207995] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3708.403296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3710.951392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3710.954257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.960546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3710.980826] Lustre: Skipped 2 previous similar messages [ 3716.068815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3716.077473] Lustre: Skipped 2 previous similar messages [ 3721.189027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3721.197130] Lustre: Skipped 1 previous similar message [ 3724.256601] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3724.369193] Lustre: server umount lustre-OST0000 complete [ 3728.864534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3728.868977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3731.425358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3731.434859] Lustre: Skipped 2 previous similar messages [ 3733.488893] Lustre: server umount lustre-MDT0000 complete [ 3736.546373] LustreError: 119662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3736.561993] LustreError: 119662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3736.634133] LustreError: 120449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747548 with bad export cookie 10176061776268993726 [ 3736.637783] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3736.640815] LustreError: 120449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3736.907753] Lustre: server umount lustre-MDT0001 complete [ 3744.099853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3744.399869] LustreError: 123839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3744.484211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3747.368131] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3749.857600] LustreError: 123840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3753.555596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.204591] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3765.253224] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3770.941220] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3776.592925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3781.696894] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3787.873787] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3797.603384] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3804.640529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3804.648577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3804.664090] Lustre: Skipped 1 previous similar message [ 3804.669968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3806.926855] Lustre: server umount lustre-OST0000 complete [ 3810.275941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3810.285340] LustreError: Skipped 1 previous similar message [ 3824.097189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3824.230414] Lustre: server umount lustre-MDT0000 complete [ 3826.145311] LustreError: 123839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3826.158396] LustreError: 123839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3827.294813] LustreError: 123820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747639 with bad export cookie 10176061776268994608 [ 3827.301752] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3827.303158] LustreError: 123820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3827.602713] Lustre: server umount lustre-MDT0001 complete [ 3835.297074] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 3837.242104] Key type lgssc unregistered [ 3837.408619] LNet: 127473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3837.415612] LNetError: 127473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3837.433115] LNet: Removed LNI 192.168.206.136@tcp [ 3838.424181] Key type .llcrypt unregistered [ 3838.426815] Key type ._llcrypt unregistered [ 3853.983988] Key type ._llcrypt registered [ 3853.985454] Key type .llcrypt registered [ 3854.055924] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 3867.473887] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3868.262519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3868.281678] alg: No test for adler32 (adler32-zlib) [ 3869.229024] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 3869.411726] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 3871.064464] Key type lgssc registered [ 3872.090996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3901.629271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3901.668271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3902.872501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3902.922279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3903.003547] Lustre: lustre-MDT0000: new disk, initializing [ 3903.060423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3903.073364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3906.705586] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3915.689396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.781869] Lustre: 130624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3915.820822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3915.823680] Lustre: Skipped 1 previous similar message [ 3915.879336] Lustre: lustre-MDT0001: new disk, initializing [ 3916.002616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3916.032053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3916.041664] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3920.246692] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3928.471849] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3933.298828] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3939.122100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3939.317595] Lustre: lustre-OST0000: new disk, initializing [ 3939.319817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3939.324504] Lustre: 132588:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3939.391136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3945.366519] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3945.556409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3945.571246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3945.667084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3954.520964] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3959.045800] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3959.245769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3963.097247] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3963.245779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3966.436307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3966.438936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.459249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3966.463912] Lustre: Skipped 1 previous similar message [ 3971.553449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3971.559042] Lustre: Skipped 1 previous similar message [ 3976.675194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3976.682525] Lustre: Skipped 1 previous similar message [ 3979.231296] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3979.374336] Lustre: server umount lustre-OST0000 complete [ 3982.818985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3982.827183] LustreError: Skipped 1 previous similar message [ 3982.832522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.846089] Lustre: Skipped 1 previous similar message [ 3982.850490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3986.912412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3986.920995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.152254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3997.154877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.167862] Lustre: Skipped 4 previous similar messages [ 3997.320257] Lustre: server umount lustre-MDT0000 complete [ 4000.665500] LustreError: 130616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747812 with bad export cookie 2497942025772168361 [ 4000.668225] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4000.677260] LustreError: 130616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4001.120461] Lustre: server umount lustre-MDT0001 complete [ 4008.556625] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:50:18 (1786747818) [ 4014.825281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4015.252356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4018.926338] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4020.712705] LustreError: 135005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4020.727209] LustreError: 135005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4025.836586] LustreError: 135004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4026.109162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4026.412338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4030.412222] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4037.794557] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4043.607496] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4045.792327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.801732] Lustre: Skipped 1 previous similar message [ 4045.806487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4045.809494] Lustre: Skipped 1 previous similar message [ 4051.807290] Lustre: server umount lustre-MDT0000 complete [ 4051.943618] LustreError: 135004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.958513] LustreError: 135004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4055.525884] LustreError: 134985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747867 with bad export cookie 2497942025772169243 [ 4055.532713] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4055.550444] LustreError: 134985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4055.953310] Lustre: server umount lustre-MDT0001 complete [ 4064.412754] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:51:14 (1786747874) [ 4071.402549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4086.943632] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4092.063450] LustreError: 137421:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.136@tcp: failed processing log, type 4: rc = -110 [ 4117.730174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4123.828325] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4131.137601] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4132.919526] Lustre: server umount lustre-OST0000 complete [ 4140.664519] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:52:30 (1786747950) [ 4146.827173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4147.131102] LustreError: 138865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.180759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.069763] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4152.289084] LustreError: 138866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4158.861425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.864136] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4170.180847] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4175.292675] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4181.127747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4181.478113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.482093] Lustre: Skipped 1 previous similar message [ 4187.537906] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4194.824818] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4201.959721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.965271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4201.980262] Lustre: Skipped 2 previous similar messages [ 4201.988300] Lustre: Skipped 4 previous similar messages [ 4207.923720] Lustre: server umount lustre-MDT0000 complete [ 4211.519949] LustreError: 138846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748023 with bad export cookie 2497942025772169908 [ 4211.522176] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4211.530993] LustreError: 138846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4211.780554] Lustre: server umount lustre-MDT0001 complete [ 4225.747458] Lustre: server umount lustre-OST0000 complete [ 4232.737338] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:54:02 (1786748042) [ 4238.299898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.615649] LustreError: 142518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4238.625123] LustreError: 142518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4242.360726] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4249.732349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.041120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4250.051550] Lustre: Skipped 1 previous similar message [ 4253.675578] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4261.383760] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4267.333990] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4273.217859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4279.432728] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4283.834441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4287.056851] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4291.147358] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4291.366192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4295.311834] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4295.645582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4299.236647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.239326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4299.256296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4299.271972] Lustre: Skipped 4 previous similar messages [ 4303.963929] Lustre: server umount lustre-OST0000 complete [ 4313.594732] Lustre: server umount lustre-MDT0000 complete [ 4314.593797] LustreError: 142518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4314.615977] LustreError: 142518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4316.736129] LustreError: 142498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748128 with bad export cookie 2497942025772170979 [ 4316.742382] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4316.751178] LustreError: 142498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4317.000735] Lustre: server umount lustre-MDT0001 complete [ 4324.528612] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:55:34 (1786748134) [ 4329.989826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4345.504948] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4350.624158] LustreError: 146680:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.136@tcp: failed processing log, type 4: rc = -110 [ 4382.192842] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4389.977621] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4396.305597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4396.715137] LustreError: 146705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4396.802894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4396.812164] Lustre: Skipped 2 previous similar messages [ 4396.875678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4401.366591] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4409.127227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.518476] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4422.653382] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4429.364874] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4433.573384] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4433.820584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4437.598821] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4437.824898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4440.034346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4440.047715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.075768] Lustre: Skipped 3 previous similar messages [ 4440.079073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4440.082842] Lustre: Skipped 2 previous similar messages [ 4445.633420] LustreError: 149828:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (00000000d527c362) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4447.679570] Lustre: server umount lustre-MDT0000 complete [ 4450.688718] LustreError: 146687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748262 with bad export cookie 2497942025772172001 [ 4450.712277] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4450.959357] Lustre: server umount lustre-MDT0001 complete [ 4464.693087] Lustre: server umount lustre-OST0000 complete [ 4471.603818] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:58:01 (1786748281) [ 4477.692033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4493.279887] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4498.399758] LustreError: 150875:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.136@tcp: failed processing log, type 4: rc = -110 [ 4529.343521] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4536.452578] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4541.954854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4542.283759] LustreError: 150900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4542.296714] LustreError: 150900:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4542.400615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4545.992423] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4553.569048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.551476] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4564.206724] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4569.715888] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4575.364094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4575.496959] Lustre: lustre-OST0001: new disk, initializing [ 4575.499723] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4575.504413] Lustre: 153746:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4580.126405] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4580.934824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4580.940116] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4581.087579] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4588.469561] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4592.461024] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4592.732143] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4598.258417] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4598.458202] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4601.853022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4601.872402] Lustre: Skipped 3 previous similar messages [ 4606.714137] Lustre: server umount lustre-OST0000 complete [ 4612.070461] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4612.089085] LustreError: Skipped 1 previous similar message [ 4616.847179] Lustre: server umount lustre-OST0001 complete [ 4620.256584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4626.518377] Lustre: server umount lustre-MDT0000 complete [ 4630.251625] LustreError: 150882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748442 with bad export cookie 2497942025772172953 [ 4630.256104] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4630.266152] LustreError: 150882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4630.645681] Lustre: server umount lustre-MDT0001 complete [ 4655.202928] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 19:01:04 (1786748464) [ 4657.016247] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4658.784746] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4660.894718] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 19:01:10 (1786748470) [ 4667.745950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4667.971383] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4668.007601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4668.197989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4668.205746] Lustre: Skipped 5 previous similar messages [ 4672.806889] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4682.032529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.126629] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4682.176945] Lustre: 157318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4687.433831] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4695.060472] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4700.965550] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4707.739954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4707.956768] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4714.555260] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4716.194171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4722.260968] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4727.635145] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4727.877835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4733.034896] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4733.262574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4736.482038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4736.496861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.517383] Lustre: Skipped 4 previous similar messages [ 4736.531290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4736.548284] Lustre: Skipped 10 previous similar messages [ 4741.642248] Lustre: server umount lustre-OST0000 complete [ 4764.719873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4771.690908] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4776.296322] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4776.305700] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4776.317155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4778.779118] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4783.876295] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4784.111291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4789.626529] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4789.894871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4795.360441] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d07067fa800.ost_server_uuid 50 [ 4796.666651] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d07067fa800.ost_server_uuid in FULL state after 0 sec [ 4796.832253] LustreError: 158778:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.36@tcp) returned error from blocking AST (req@00000000e2fd3947 x1873540736607232 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000292697f8/0x22aa7909abab8ff3 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.36@tcp remote: 0xd9fe74a811ce3987 expref: 5 pid: 159475 timeout: 4896 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4796.879946] LustreError: lustre-MDT0000: A client on nid 192.168.206.36@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4796.890617] LustreError: 157313:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.36@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000292697f8/0x22aa7909abab8ff3 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.36@tcp remote: 0xd9fe74a811ce3987 expref: 6 pid: 159475 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4802.024663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4802.038803] LustreError: Skipped 3 previous similar messages [ 4807.067296] Lustre: server umount lustre-OST0000 complete [ 4816.794702] Lustre: server umount lustre-MDT0000 complete [ 4817.379725] LustreError: 157323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4817.403440] LustreError: 157323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4820.138082] LustreError: 157309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748631 with bad export cookie 2497942025772174136 [ 4820.145681] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4820.153955] LustreError: 157309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4826.673487] Lustre: server umount lustre-MDT0001 complete [ 4837.474752] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 4840.568025] Key type lgssc unregistered [ 4840.812778] LNet: 163435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4840.819025] LNetError: 163435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4840.840977] LNet: Removed LNI 192.168.206.136@tcp [ 4841.615324] Key type .llcrypt unregistered [ 4841.619343] Key type ._llcrypt unregistered [ 4863.760683] Key type ._llcrypt registered [ 4863.763126] Key type .llcrypt registered [ 4863.867133] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 4864.717500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4864.810866] alg: No test for adler32 (adler32-zlib) [ 4865.835343] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 4866.038450] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 4867.703216] Key type lgssc registered [ 4868.695462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4875.423594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4875.441457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4876.734536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4880.929703] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4887.016117] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4887.032228] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4888.910347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4889.154822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4893.343063] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4901.486443] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4906.766987] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4913.611242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4913.948867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4920.560888] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4922.170850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4929.190611] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4939.581341] Lustre: Failing over lustre-MDT0000 [ 4939.815706] Lustre: server umount lustre-MDT0000 complete [ 4940.261168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4940.269930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.296899] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4943.849277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.850711] LustreError: 165576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4943.865243] Lustre: Skipped 1 previous similar message [ 4943.883016] LustreError: 165576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4948.971286] LustreError: 164849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4948.982924] LustreError: 164849:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4952.450629] LustreError: 164829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.36@tcp arrived at 1786748764 with bad export cookie 5802318342761994938 [ 4952.468875] LustreError: 164829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4954.105838] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4954.131418] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4959.241459] Lustre: 164159:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786748755/real 1786748755] req@00000000147c0413 x1873541780762624/t0(0) o400->MGC192.168.206.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786748771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4959.272837] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4960.878128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4964.323921] LustreError: 164843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4964.349448] LustreError: 164843:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4969.451404] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5085f9ca6e0fdb23 [ 4969.488337] Lustre: MGC192.168.206.136@tcp: Connection restored to 0@lo (at 0@lo) [ 4969.773534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4969.834741] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4974.580673] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4975.078524] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4975.101648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4977.390656] Lustre: lustre-MDT0000: Denying connection for new client 6c7fad46-2824-4b4e-89c5-3d76e4b26fa8 (at 192.168.206.36@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4989.922295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4989.938574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.976459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4992.496124] Lustre: server umount lustre-OST0000 complete [ 4996.165836] LustreError: 168586:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4996.171833] Lustre: 167936:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4996.182271] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4996.193206] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4996.204188] LustreError: 167936:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a41b30bc x1873541780782592/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4996.214062] LustreError: 167936:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4996.224341] LustreError: 167936:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4996.232295] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 4996.597377] Lustre: server umount lustre-MDT0000 complete [ 5000.516848] LustreError: 167428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748812 with bad export cookie 5802318342761995043 [ 5000.525154] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5000.534832] LustreError: 165576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5000.545742] LustreError: 165576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5002.954768] Lustre: server umount lustre-MDT0001 complete [ 5014.110271] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 5016.420690] Key type lgssc unregistered [ 5016.684937] LNet: 169341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5016.692035] LNetError: 169341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5017.702728] LNet: Removed LNI 192.168.206.136@tcp [ 5018.533447] Key type .llcrypt unregistered [ 5018.540384] Key type ._llcrypt unregistered [ 5040.653410] Key type ._llcrypt registered [ 5040.655646] Key type .llcrypt registered [ 5040.749099] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 5042.143796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5042.162096] alg: No test for adler32 (adler32-zlib) [ 5043.324642] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 5043.622708] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 5045.311178] Key type lgssc registered [ 5046.496470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5054.985563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5055.041049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5056.696791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5062.395334] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5067.243066] LustreError: 170754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5067.261433] LustreError: 170754:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5070.700529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.079919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5075.296740] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5082.701498] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5089.697995] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5097.344909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5097.880478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.026558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5105.005592] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5112.883157] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5120.488706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5120.496424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5120.509048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5120.514619] Lustre: Skipped 1 previous similar message [ 5120.531752] Lustre: Skipped 1 previous similar message [ 5122.344943] Lustre: server umount lustre-OST0000 complete [ 5127.653649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5127.661592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.685688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5130.724397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5130.729389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5134.143846] Lustre: server umount lustre-MDT0000 complete [ 5135.843123] LustreError: 170754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5135.886307] LustreError: 170754:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5139.043337] LustreError: 170736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748950 with bad export cookie 9815835484197440918 [ 5139.045660] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5139.058743] LustreError: 170736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5139.641856] Lustre: server umount lustre-MDT0001 complete [ 5154.259918] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 5157.150684] Key type lgssc unregistered [ 5157.513728] LNet: 174288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5157.523987] LNetError: 174288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5157.547497] LNet: Removed LNI 192.168.206.136@tcp [ 5158.626390] Key type .llcrypt unregistered [ 5158.628049] Key type ._llcrypt unregistered [ 5202.699803] Key type ._llcrypt registered [ 5202.702454] Key type .llcrypt registered [ 5202.869691] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 5204.579533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5204.623575] alg: No test for adler32 (adler32-zlib) [ 5206.007545] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 5206.391191] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 5208.183153] Key type lgssc registered [ 5209.399084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5217.237688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5217.262314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.714970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5222.874147] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5229.035047] LustreError: 175992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5229.048144] LustreError: 175992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5233.014070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5233.562576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5241.678479] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5253.011169] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5260.732473] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5268.199614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5268.638354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5276.477378] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5284.920883] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5289.150054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5299.680362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5299.686770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5299.690278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5299.697298] Lustre: Skipped 1 previous similar message [ 5302.212869] Lustre: server umount lustre-OST0000 complete [ 5309.920777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5309.939080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5309.953825] Lustre: Skipped 2 previous similar messages [ 5313.266787] Lustre: server umount lustre-MDT0000 complete [ 5315.040809] LustreError: 175986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5315.052808] LustreError: 175986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5316.936312] LustreError: 175972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749128 with bad export cookie 4670925963505011123 [ 5316.940188] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5316.958054] LustreError: 175972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5317.330867] Lustre: server umount lustre-MDT0001 complete [ 5327.960624] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 5331.253733] Key type lgssc unregistered [ 5331.587791] LNet: 179528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5331.608181] LNetError: 179528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5331.625932] LNet: Removed LNI 192.168.206.136@tcp [ 5332.506235] Key type .llcrypt unregistered [ 5332.508642] Key type ._llcrypt unregistered [ 5355.468481] Key type ._llcrypt registered [ 5355.472768] Key type .llcrypt registered [ 5355.571480] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 5356.698174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5356.727898] alg: No test for adler32 (adler32-zlib) [ 5357.724731] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 5357.923538] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 5359.575526] Key type lgssc registered [ 5360.623806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5370.914803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5370.939951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5372.301908] Lustre: *** cfs_fail_loc=135, val=0*** [ 5372.382752] LustreError: 181053:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5372.390392] LustreError: 181053:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.136@tcp: cfg command failed: rc = -2 [ 5372.395722] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5372.395722] [ 5372.401714] LustreError: MGC192.168.206.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5372.411454] LustreError: 181015:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5372.415684] LustreError: 181015:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5372.424355] LustreError: 181015:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5372.430842] LustreError: 181015:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5372.521674] Lustre: server umount lustre-MDT0000 complete [ 5372.525348] LustreError: 181015:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5378.903539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.212778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5383.274623] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5388.775300] LustreError: 181423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5388.794550] LustreError: 181423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5392.158385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5392.604681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5397.432185] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5404.247354] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5410.976269] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5414.381956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5414.397056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5414.409656] Lustre: Skipped 1 previous similar message [ 5417.506175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5417.528310] Lustre: Skipped 1 previous similar message [ 5420.080878] Lustre: server umount lustre-MDT0000 complete [ 5422.560758] LustreError: 181424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.587101] LustreError: 181424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5424.885756] LustreError: 181405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749236 with bad export cookie 2860319942113246429 [ 5424.894646] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5424.900393] LustreError: 181405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5425.201807] Lustre: server umount lustre-MDT0001 complete [ 5440.592669] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 5443.454994] Key type lgssc unregistered [ 5443.852659] LNet: 183750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5443.859111] LNetError: 183750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5443.875751] LNet: Removed LNI 192.168.206.136@tcp [ 5444.886110] Key type .llcrypt unregistered [ 5444.887733] Key type ._llcrypt unregistered [ 5470.082378] Key type ._llcrypt registered [ 5470.085507] Key type .llcrypt registered [ 5470.227942] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 5470.599665] Key type .llcrypt unregistered [ 5470.603938] Key type ._llcrypt unregistered [ 5488.328925] Key type ._llcrypt registered [ 5488.332645] Key type .llcrypt registered [ 5488.497569] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 5489.986766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5490.007751] alg: No test for adler32 (adler32-zlib) [ 5491.248206] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 5491.554403] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 5493.249646] Key type lgssc registered [ 5494.349157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5504.035388] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5504.079602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5519.711428] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5524.836657] LustreError: 185886:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.136@tcp: failed processing log, type 4: rc = -110 [ 5551.583589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5558.027423] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5565.604948] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5572.383662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5572.958409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5573.014111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5577.351641] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5578.210405] LustreError: 185912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5578.224094] LustreError: 185912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5583.332115] LustreError: 185911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5585.646196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5586.164969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5590.598049] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5599.092979] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5605.970837] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5609.533438] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5622.240642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5622.251347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.265260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5623.267685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5623.277203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.384940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5628.396741] Lustre: Skipped 2 previous similar messages [ 5633.504969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5633.515985] Lustre: Skipped 1 previous similar message [ 5636.063150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5636.510302] Lustre: server umount lustre-OST0000 complete [ 5643.747625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5643.788454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5643.804059] Lustre: Skipped 1 previous similar message [ 5649.407709] Lustre: server umount lustre-MDT0000 complete [ 5653.989995] LustreError: 188356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5654.019330] LustreError: 188356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5654.567676] LustreError: 187666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749466 with bad export cookie 7002830776409144982 [ 5654.569866] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5654.583968] LustreError: 187666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5654.835658] Lustre: server umount lustre-MDT0001 complete [ 5668.413718] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 5670.980155] Key type lgssc unregistered [ 5671.283715] LNet: 189859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5671.293729] LNetError: 189859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5671.313219] LNet: Removed LNI 192.168.206.136@tcp [ 5672.292585] Key type .llcrypt unregistered [ 5672.295239] Key type ._llcrypt unregistered [ 5694.149163] Key type ._llcrypt registered [ 5694.151404] Key type .llcrypt registered [ 5694.224695] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 5694.852191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5694.923989] alg: No test for adler32 (adler32-zlib) [ 5695.920280] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 5696.105771] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 5697.799147] Key type lgssc registered [ 5698.754233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5706.211377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5706.223207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5707.687112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5712.752717] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5717.986251] LustreError: 191253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5718.006308] LustreError: 191253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5720.652565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5721.008046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5725.875650] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5734.234809] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5740.537484] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5747.698326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5748.006775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5753.157614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5754.617437] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5763.158520] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5769.062936] Lustre: Failing over lustre-MDT0000 [ 5769.469647] Lustre: server umount lustre-MDT0000 complete [ 5770.726794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.740786] LustreError: 191249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.754579] Lustre: Skipped 2 previous similar messages [ 5770.771729] LustreError: 191249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5775.848678] LustreError: 191253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.873338] LustreError: 191253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5780.962436] LustreError: 191980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5780.996606] LustreError: 191980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5786.087301] LustreError: 191254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5786.122924] LustreError: 191254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5787.104119] Lustre: 190564:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749582/real 1786749582] req@00000000b166e175 x1873542651078656/t0(0) o400->MGC192.168.206.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786749598 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5787.150118] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.488412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5796.326240] LustreError: 190561:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b8ac7d41 x1873542651083776/t0(0) o250->MGC192.168.206.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5796.379641] LustreError: 191253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5796.415855] LustreError: 191253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 5796.791614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5796.876186] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5797.473573] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5801.969726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5801.989753] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5802.043782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5802.459899] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5808.502299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5814.496515] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5821.920346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5821.929718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5821.943351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.444841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.454288] Lustre: Skipped 1 previous similar message [ 5823.095272] Lustre: server umount lustre-OST0000 complete [ 5827.554187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5827.556122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.559826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5827.581831] Lustre: Skipped 2 previous similar messages [ 5832.712019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5832.713817] LustreError: 191249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5832.723074] Lustre: Skipped 1 previous similar message [ 5832.894746] Lustre: server umount lustre-MDT0000 complete [ 5836.789386] LustreError: 191987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749648 with bad export cookie 16113508175286967102 [ 5836.790330] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.804030] LustreError: 191987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5837.202712] Lustre: server umount lustre-MDT0001 complete [ 5847.968290] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 5850.783214] Key type lgssc unregistered [ 5851.064338] LNet: 196094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5851.070947] LNetError: 196094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5852.141263] LNet: Removed LNI 192.168.206.136@tcp [ 5853.122932] Key type .llcrypt unregistered [ 5853.129167] Key type ._llcrypt unregistered [ 5876.759155] Key type ._llcrypt registered [ 5876.760858] Key type .llcrypt registered [ 5876.857598] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 5878.060026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5878.118111] alg: No test for adler32 (adler32-zlib) [ 5879.118885] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 5879.355336] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 5881.015163] Key type lgssc registered [ 5882.078390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5890.032655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5890.054406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5891.521324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5895.891294] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5901.800608] LustreError: 197508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5901.827050] LustreError: 197508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5902.860652] LustreError: 197509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5905.411803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5905.830640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5910.161084] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5919.219522] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5925.811385] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5932.225192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5932.621143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5937.779800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5938.426977] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5946.114699] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5949.913716] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5953.271699] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5967.853623] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5973.984449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5973.994595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5974.037483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5977.056121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5977.058100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5977.070868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.121423] Lustre: Skipped 1 previous similar message [ 5979.747729] Lustre: server umount lustre-OST0000 complete [ 5987.297455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5987.303841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5987.338341] Lustre: Skipped 1 previous similar message [ 5989.767188] Lustre: server umount lustre-MDT0000 complete [ 5992.420777] LustreError: 197509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5992.453728] LustreError: 197509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5994.509920] LustreError: 200045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749806 with bad export cookie 3344318016638751547 [ 5994.520147] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5994.520907] LustreError: 200045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5994.875815] Lustre: server umount lustre-MDT0001 complete [ 6009.006420] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 6011.588820] Key type lgssc unregistered [ 6011.812968] LNet: 201357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6011.820198] LNetError: 201357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6011.847174] LNet: Removed LNI 192.168.206.136@tcp [ 6012.707523] Key type .llcrypt unregistered [ 6012.710524] Key type ._llcrypt unregistered [ 6035.432366] Key type ._llcrypt registered [ 6035.436768] Key type .llcrypt registered [ 6035.566415] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 6036.855665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6036.968116] alg: No test for adler32 (adler32-zlib) [ 6038.081871] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 6038.289985] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 6039.935244] Key type lgssc registered [ 6041.022498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6049.394569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6049.411676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.839581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6056.349868] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6061.033911] LustreError: 202623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.046906] LustreError: 202623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6064.816421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6065.277227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6069.777738] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6076.797590] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6082.642515] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6088.962416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6089.299472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6094.843967] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6095.471883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6102.150415] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6107.453307] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6121.296154] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6131.716263] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6137.563138] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6147.042277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6147.043053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.055764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6147.072263] Lustre: Skipped 1 previous similar message [ 6150.966933] Lustre: server umount lustre-OST0000 complete [ 6157.291697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.295026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6157.315579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6157.318029] LustreError: Skipped 1 previous similar message [ 6157.333110] Lustre: Skipped 2 previous similar messages [ 6162.401425] LustreError: 204845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.425953] LustreError: 204845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6163.153331] Lustre: server umount lustre-MDT0000 complete [ 6166.511337] LustreError: 202618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6166.523158] LustreError: 202618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6167.409582] LustreError: 202606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749979 with bad export cookie 3310401374336818014 [ 6167.415408] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6167.430151] LustreError: 202606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6167.756228] Lustre: server umount lustre-MDT0001 complete [ 6179.769890] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 6182.957661] Key type lgssc unregistered [ 6183.308507] LNet: 207551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6183.318941] LNetError: 207551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6183.340992] LNet: Removed LNI 192.168.206.136@tcp [ 6184.223560] Key type .llcrypt unregistered [ 6184.226271] Key type ._llcrypt unregistered [ 6209.742966] Key type ._llcrypt registered [ 6209.746814] Key type .llcrypt registered [ 6209.873981] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 6210.922088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6211.011100] alg: No test for adler32 (adler32-zlib) [ 6212.245837] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 6212.602313] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 6214.279276] Key type lgssc registered [ 6215.347729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6223.062582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6223.098728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.578240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.370147] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6235.105909] LustreError: 208975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.121250] LustreError: 208975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6237.577253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6237.966087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6242.826217] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6250.764347] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6257.207969] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6264.866546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6265.289587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6269.427338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6272.351937] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6280.564544] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6298.082466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.083442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6298.102823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6298.120115] Lustre: Skipped 1 previous similar message [ 6302.972829] Lustre: server umount lustre-OST0000 complete [ 6308.320775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6308.326826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6308.343471] Lustre: Skipped 1 previous similar message [ 6308.349736] Lustre: Skipped 1 previous similar message [ 6313.442704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6313.452212] Lustre: Skipped 2 previous similar messages [ 6313.866798] Lustre: server umount lustre-MDT0000 complete [ 6317.537886] LustreError: 209705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6317.555803] LustreError: 209705:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6318.041841] LustreError: 208957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750129 with bad export cookie 2167474479976129368 [ 6318.055583] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6318.505908] Lustre: server umount lustre-MDT0001 complete [ 6330.654565] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 6333.557365] Key type lgssc unregistered [ 6333.986173] LNet: 212661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6333.993759] LNetError: 212661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6334.008795] LNet: Removed LNI 192.168.206.136@tcp [ 6335.118656] Key type .llcrypt unregistered [ 6335.122369] Key type ._llcrypt unregistered [ 6354.360493] Key type ._llcrypt registered [ 6354.363194] Key type .llcrypt registered [ 6354.484429] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 6355.958537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6355.997955] alg: No test for adler32 (adler32-zlib) [ 6357.140793] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 6357.453941] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 6359.151695] Key type lgssc registered [ 6360.501268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6370.982471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6370.996126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.637908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6377.366688] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6383.072133] LustreError: 213926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.095442] LustreError: 213926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6387.577366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6388.015558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6393.138954] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6401.545732] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6408.079951] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6417.255918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6417.708641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6418.736860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6426.865738] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6436.583422] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6453.729638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6453.737296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6453.745572] Lustre: Skipped 1 previous similar message [ 6453.747084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6453.750683] Lustre: Skipped 1 previous similar message [ 6458.853911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6458.873767] Lustre: Skipped 1 previous similar message [ 6459.711299] Lustre: server umount lustre-OST0000 complete [ 6463.968570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6463.975662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6463.985838] Lustre: Skipped 1 previous similar message [ 6469.089223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6469.099384] Lustre: Skipped 2 previous similar messages [ 6470.141179] Lustre: server umount lustre-MDT0000 complete [ 6474.073496] LustreError: 213907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750285 with bad export cookie 311470227489704859 [ 6474.077569] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6474.085365] LustreError: 213907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6474.535427] Lustre: server umount lustre-MDT0001 complete [ 6487.331702] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 6489.873395] Key type lgssc unregistered [ 6490.151319] LNet: 217617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6490.161173] LNetError: 217617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6490.177299] LNet: Removed LNI 192.168.206.136@tcp [ 6490.997729] Key type .llcrypt unregistered [ 6491.001771] Key type ._llcrypt unregistered [ 6509.019983] Key type ._llcrypt registered [ 6509.022737] Key type .llcrypt registered [ 6509.134475] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 6510.523669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6510.575142] alg: No test for adler32 (adler32-zlib) [ 6511.826580] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 6512.092806] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 6513.791200] Key type lgssc registered [ 6515.181293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6523.335951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6523.350169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6525.001066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6530.387764] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6535.142245] LustreError: 218901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.179285] LustreError: 218901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6539.639769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6540.053348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6544.989944] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6553.560193] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6559.196963] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6565.846407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6566.313110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6569.405234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6573.739974] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6581.366286] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6590.667675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6590.728257] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6590.934727] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6597.345682] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6604.804699] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6606.065930] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6617.947381] Lustre: Permanently deactivating lustre-OST0001 [ 6617.958933] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6624.815533] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6632.786344] Lustre: Permanently reactivating lustre-OST0001 [ 6632.808277] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6632.819359] Lustre: Skipped 2 previous similar messages [ 6638.686832] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6638.706027] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6638.714218] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6638.743596] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6638.791560] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6644.191945] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6644.206928] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6644.208053] LustreError: Skipped 1 previous similar message [ 6644.214881] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6644.230083] Lustre: Skipped 2 previous similar messages [ 6644.241400] Lustre: Skipped 1 previous similar message [ 6646.262903] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6646.276270] Lustre: Skipped 1 previous similar message [ 6649.097914] Lustre: server umount lustre-OST0001 complete [ 6651.372977] LustreError: 220723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6651.391121] LustreError: 220723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6656.480278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6656.483222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6656.483804] LustreError: 221433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6656.488434] LustreError: Skipped 1 previous similar message [ 6656.489117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6656.535159] Lustre: Skipped 1 previous similar message [ 6658.941786] Lustre: server umount lustre-OST0000 complete [ 6666.729172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6666.741697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6666.748637] Lustre: Skipped 2 previous similar messages [ 6669.731294] Lustre: server umount lustre-MDT0000 complete [ 6671.843525] LustreError: 218896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6671.869104] LustreError: 218896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6673.871049] LustreError: 218880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750485 with bad export cookie 2661944795689681772 [ 6673.877561] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6673.883425] LustreError: 218880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6674.725457] Lustre: server umount lustre-MDT0001 complete [ 6687.473646] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 6690.191545] Key type lgssc unregistered [ 6690.456877] LNet: 223947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6690.463017] LNetError: 223947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6690.487762] LNet: Removed LNI 192.168.206.136@tcp [ 6691.287543] Key type .llcrypt unregistered [ 6691.289450] Key type ._llcrypt unregistered [ 6737.816591] Key type ._llcrypt registered [ 6737.820406] Key type .llcrypt registered [ 6737.983410] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 6739.476390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6739.562529] alg: No test for adler32 (adler32-zlib) [ 6740.757410] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 6741.057596] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 6742.783412] Key type lgssc registered [ 6744.024429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6751.866920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6751.884985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.157187] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6753.185880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6753.347528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6758.847355] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6767.586773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.723093] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6767.765456] Lustre: 226155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6768.059700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6772.840871] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6780.911438] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6787.358130] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6794.348125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6794.590051] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6794.812587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6801.867843] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6804.251376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6809.815739] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6813.899633] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6817.368916] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6824.168067] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6832.108222] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6842.590994] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6863.509255] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6863.518689] Lustre: Skipped 2 previous similar messages [ 6899.623934] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6899.632660] Lustre: Skipped 4 previous similar messages [ 6964.634411] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6964.640149] Lustre: Skipped 7 previous similar messages [ 6983.650824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6983.653126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6983.662740] Lustre: Skipped 1 previous similar message [ 6983.679708] Lustre: Skipped 1 previous similar message [ 6985.445936] Lustre: server umount lustre-OST0000 complete [ 6993.380763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6993.390456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6993.408555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6999.009251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6999.026074] Lustre: Skipped 2 previous similar messages [ 7004.127154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7004.131846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7004.147571] Lustre: Skipped 1 previous similar message [ 7004.446973] Lustre: server umount lustre-MDT0000 complete [ 7008.250550] LustreError: 229495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750820 with bad export cookie 3500367489330787726 [ 7008.254291] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7008.269990] LustreError: 229495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7008.684900] Lustre: server umount lustre-MDT0001 complete [ 7020.868677] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 7024.062940] Key type lgssc unregistered [ 7024.403605] LNet: 230946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7024.420320] LNetError: 230946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7024.438928] LNet: Removed LNI 192.168.206.136@tcp [ 7025.190414] Key type .llcrypt unregistered [ 7025.192575] Key type ._llcrypt unregistered [ 7048.864240] Key type ._llcrypt registered [ 7048.870588] Key type .llcrypt registered [ 7048.988299] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7050.261798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7050.297265] alg: No test for adler32 (adler32-zlib) [ 7051.462861] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 7051.854608] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 7053.567547] Key type lgssc registered [ 7054.952488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7063.634275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7063.653155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7065.235488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7070.068846] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7074.791886] LustreError: 232360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7074.810520] LustreError: 232360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7079.770262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7079.905370] LustreError: 232361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7080.195335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7085.085399] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7093.604449] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7100.958982] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7108.604804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7108.903732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7109.942199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7115.182822] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7122.974305] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7139.818503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7139.831085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.833081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7139.846166] Lustre: Skipped 1 previous similar message [ 7139.863715] Lustre: Skipped 1 previous similar message [ 7140.547541] Lustre: server umount lustre-OST0000 complete [ 7144.929794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.943292] Lustre: Skipped 1 previous similar message [ 7144.944471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7150.053975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7150.059351] Lustre: Skipped 2 previous similar messages [ 7150.596967] Lustre: server umount lustre-MDT0000 complete [ 7154.213606] LustreError: 232343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750965 with bad export cookie 5136601347272531220 [ 7154.222450] LustreError: 232343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7154.222664] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7154.617381] Lustre: server umount lustre-MDT0001 complete [ 7164.891202] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 7167.521327] Key type lgssc unregistered [ 7167.813107] LNet: 236054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7167.818882] LNetError: 236054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7167.834352] LNet: Removed LNI 192.168.206.136@tcp [ 7168.546433] Key type .llcrypt unregistered [ 7168.550813] Key type ._llcrypt unregistered [ 7191.484844] Key type ._llcrypt registered [ 7191.486609] Key type .llcrypt registered [ 7191.578418] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7192.460616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7192.500419] alg: No test for adler32 (adler32-zlib) [ 7193.604136] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 7193.897937] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 7195.607177] Key type lgssc registered [ 7196.660621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7466.074593] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 7469.024432] Key type lgssc unregistered [ 7469.464290] LNet: 238080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7469.477199] LNetError: 238080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7469.507661] LNet: Removed LNI 192.168.206.136@tcp [ 7470.583285] Key type .llcrypt unregistered [ 7470.587292] Key type ._llcrypt unregistered [ 7494.728359] Key type ._llcrypt registered [ 7494.731566] Key type .llcrypt registered [ 7494.832920] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 7509.364800] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7510.744939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7510.775526] alg: No test for adler32 (adler32-zlib) [ 7511.845198] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 7512.114393] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 7513.847157] Key type lgssc registered [ 7514.976216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7548.038406] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7273 sec ========== 19:49:17 (1786751357) [ 7550.251846] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:49:19 (1786751359) === [ 7554.316223] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:49:23 (1786751363) === [ 7595.210845] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 7597.812279] Key type lgssc unregistered [ 7598.193314] LNet: 243318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7598.203820] LNetError: 243318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7598.213702] LNet: Removed LNI 192.168.206.136@tcp [ 7598.969895] Key type .llcrypt unregistered [ 7598.971953] Key type ._llcrypt unregistered