[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505778522 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003261] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008474] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011126] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014070] *** VALIDATE selinux *** [ 0.022765] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027781] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029092] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030100] *** VALIDATE tmpfs *** [ 0.031414] *** VALIDATE proc *** [ 0.032195] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034225] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041602] debug: unmapping init [mem 0xffffffffba259000-0xffffffffba260fff] [ 0.043996] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044697] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045018] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047010] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050010] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052295] rcu: Hierarchical SRCU implementation. [ 0.054430] smp: Bringing up secondary CPUs ... [ 0.055541] x86: Booting SMP configuration: [ 0.056021] .... node #0, CPUs: #1 #2 #3 [ 0.059243] smp: Brought up 1 node, 4 CPUs [ 0.061011] smpboot: Max logical packages: 1 [ 0.062014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136018] node 0 deferred pages initialised in 71ms [ 0.139111] devtmpfs: initialized [ 0.140317] x86/mm: Memory block size: 128MB [ 0.143888] gcov: version magic: 0x41383552 [ 0.147242] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149319] pinctrl core: initialized pinctrl subsystem [ 0.150180] [ 0.151009] ************************************************************* [ 0.152012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153012] ** ** [ 0.154016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155011] ** ** [ 0.156012] ** This means that this kernel is built to expose internal ** [ 0.157012] ** IOMMU data structures, which may compromise security on ** [ 0.158013] ** your system. ** [ 0.159014] ** ** [ 0.160014] ** If you see this message and you are not debugging the ** [ 0.161017] ** kernel, report this immediately to your vendor! ** [ 0.162024] ** ** [ 0.163015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164013] ************************************************************* [ 0.165819] NET: Registered protocol family 16 [ 0.167444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176048] cpuidle: using governor menu [ 0.177819] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180468] PCI: Using configuration type 1 for base access [ 0.182127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192099] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200125] cryptd: max_cpu_qlen set to 1000 [ 0.204329] ACPI: Added _OSI(Module Device) [ 0.205012] ACPI: Added _OSI(Processor Device) [ 0.207013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213338] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.219437] ACPI: Interpreter enabled [ 0.221071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.222013] ACPI: Using IOAPIC for interrupt routing [ 0.225129] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.228453] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239399] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.247085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.252300] acpiphp: Slot [2] registered [ 0.254079] acpiphp: Slot [5] registered [ 0.255067] acpiphp: Slot [6] registered [ 0.256092] acpiphp: Slot [7] registered [ 0.257109] acpiphp: Slot [8] registered [ 0.259083] acpiphp: Slot [9] registered [ 0.260085] acpiphp: Slot [10] registered [ 0.261092] acpiphp: Slot [3] registered [ 0.262070] acpiphp: Slot [4] registered [ 0.264072] acpiphp: Slot [11] registered [ 0.265076] acpiphp: Slot [12] registered [ 0.266089] acpiphp: Slot [13] registered [ 0.267070] acpiphp: Slot [14] registered [ 0.269088] acpiphp: Slot [15] registered [ 0.270084] acpiphp: Slot [16] registered [ 0.271084] acpiphp: Slot [17] registered [ 0.272073] acpiphp: Slot [18] registered [ 0.273103] acpiphp: Slot [19] registered [ 0.275082] acpiphp: Slot [20] registered [ 0.276070] acpiphp: Slot [21] registered [ 0.277094] acpiphp: Slot [22] registered [ 0.278090] acpiphp: Slot [23] registered [ 0.280099] acpiphp: Slot [24] registered [ 0.281079] acpiphp: Slot [25] registered [ 0.282120] acpiphp: Slot [26] registered [ 0.283078] acpiphp: Slot [27] registered [ 0.285089] acpiphp: Slot [28] registered [ 0.286078] acpiphp: Slot [29] registered [ 0.287084] acpiphp: Slot [30] registered [ 0.289082] acpiphp: Slot [31] registered [ 0.290051] PCI host bridge to bus 0000:00 [ 0.291016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.293016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.295018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.297018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.299018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.304900] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.308143] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.317014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.321057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.323016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.325016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.327013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.329469] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.331886] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.334041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.336787] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.342015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.354014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.359007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.364733] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.377015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.387014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.401015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.409630] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.418014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.425014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.448014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.460000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.466012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.472015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.484015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.494000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.507014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.519015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.541016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.549554] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.558016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.569015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.583017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.594069] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.603013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.610014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.625015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.634867] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.637355] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.638265] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.640290] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.642189] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.648090] iommu: Default domain type: Passthrough [ 0.649400] SCSI subsystem initialized [ 0.651119] ACPI: bus type USB registered [ 0.653103] usbcore: registered new interface driver usbfs [ 0.655091] usbcore: registered new interface driver hub [ 0.657068] usbcore: registered new device driver usb [ 0.658106] pps_core: LinuxPPS API ver. 1 registered [ 0.659006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.662053] PTP clock support registered [ 0.664030] EDAC MC: Ver: 3.0.0 [ 0.666115] PCI: Using ACPI for IRQ routing [ 0.667764] NetLabel: Initializing [ 0.669010] NetLabel: domain hash size = 128 [ 0.671010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.673074] NetLabel: unlabeled traffic allowed by default [ 0.675111] vgaarb: loaded [ 0.677249] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.678014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.683505] clocksource: Switched to clocksource kvm-clock [ 0.786994] VFS: Disk quotas dquot_6.6.0 [ 0.788629] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.790601] *** VALIDATE ramfs *** [ 0.791672] *** VALIDATE hugetlbfs *** [ 0.793202] pnp: PnP ACPI init [ 0.795493] pnp: PnP ACPI: found 6 devices [ 0.810967] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.813830] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.815779] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.817568] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.819793] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.821855] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.824465] NET: Registered protocol family 2 [ 0.826626] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.830794] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.834136] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.838496] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.841937] TCP: Hash tables configured (established 65536 bind 65536) [ 0.844528] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.847322] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.849895] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.852549] NET: Registered protocol family 1 [ 0.855523] RPC: Registered named UNIX socket transport module. [ 0.858623] RPC: Registered udp transport module. [ 0.860405] RPC: Registered tcp transport module. [ 0.861871] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.863954] NET: Registered protocol family 44 [ 0.865291] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.867270] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.869194] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.871326] PCI: CLS 0 bytes, default 64 [ 0.872637] Unpacking initramfs... [ 2.279586] debug: unmapping init [mem 0xffff9d837cc54000-0xffff9d837ffbffff] [ 2.283403] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.285383] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.288469] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.816087] Initialise system trusted keyrings [ 2.817680] Key type blacklist registered [ 2.819608] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.828546] zbud: loaded [ 2.831828] *** VALIDATE nfs *** [ 2.833112] *** VALIDATE nfs4 *** [ 2.834687] pstore: using deflate compression [ 2.837872] Platform Keyring initialized [ 2.928351] NET: Registered protocol family 38 [ 2.930133] Key type asymmetric registered [ 2.931653] Asymmetric key parser 'x509' registered [ 2.933093] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.935940] io scheduler mq-deadline registered [ 2.937715] io scheduler kyber registered [ 2.939203] io scheduler bfq registered [ 2.940941] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.943795] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.946646] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.949360] ACPI: Power Button [PWRF] [ 2.954375] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.959452] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.971419] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.977954] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.989321] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.016900] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.049651] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.055069] Non-volatile memory driver v1.3 [ 3.056794] Linux agpgart interface v0.103 [ 3.086758] virtio_blk virtio1: [vda] 146936 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.089685] vda: detected capacity change from 0 to 75231232 [ 3.105114] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.107752] vdb: detected capacity change from 0 to 1073741824 [ 3.130445] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.133574] vdc: detected capacity change from 0 to 2621440000 [ 3.157897] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.160761] vdd: detected capacity change from 0 to 2621440000 [ 3.182509] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.185321] vde: detected capacity change from 0 to 4294967296 [ 3.200703] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.203729] vdf: detected capacity change from 0 to 4294967296 [ 3.212471] libphy: Fixed MDIO Bus: probed [ 3.218323] usbcore: registered new interface driver usbserial_generic [ 3.220777] usbserial: USB Serial support registered for generic [ 3.222933] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.227394] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.229240] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.231565] mousedev: PS/2 mouse device common for all mice [ 3.234660] rtc_cmos 00:05: RTC can wake from S4 [ 3.237672] rtc_cmos 00:05: registered as rtc0 [ 3.237943] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.239210] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.244677] intel_pstate: CPU model not supported [ 3.246756] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.250458] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.250498] hid: raw HID events driver (C) Jiri Kosina [ 3.255394] usbcore: registered new interface driver usbhid [ 3.257627] usbhid: USB HID core driver [ 3.259347] drop_monitor: Initializing network drop monitor service [ 3.261728] Initializing XFRM netlink socket [ 3.263618] NET: Registered protocol family 10 [ 3.266128] Segment Routing with IPv6 [ 3.267564] NET: Registered protocol family 17 [ 3.269384] mpls_gso: MPLS GSO support [ 3.274822] RAS: Correctable Errors collector initialized. [ 3.276915] AVX version of gcm_enc/dec engaged. [ 3.278355] AES CTR mode by8 optimization enabled [ 3.349652] sched_clock: Marking stable (3349632728, 0)->(4299520299, -949887571) [ 3.353316] registered taskstats version 1 [ 3.355364] Loading compiled-in X.509 certificates [ 3.357508] zswap: loaded using pool lzo/zbud [ 3.378665] Key type big_key registered [ 3.391918] Key type encrypted registered [ 3.393745] ima: No TPM chip found, activating TPM-bypass! [ 3.395947] ima: Allocated hash algorithm: sha1 [ 3.398081] ima: No architecture policies found [ 3.399837] evm: Initialising EVM extended attributes: [ 3.403792] evm: security.selinux [ 3.406599] evm: security.ima [ 3.407634] evm: security.capability [ 3.409025] evm: HMAC attrs: 0x1 [ 3.411247] rtc_cmos 00:05: setting system clock to 2026-08-27 15:51:47 UTC (1787845907) [ 3.417264] debug: unmapping init [mem 0xffffffffbb203000-0xffffffffbb3fffff] [ 3.420909] debug: unmapping init [mem 0xffffffffb9f82000-0xffffffffba258fff] [ 3.425120] Write protecting the kernel read-only data: 28672k [ 3.427402] debug: unmapping init [mem 0xffffffffb8603000-0xffffffffb87fffff] [ 3.429830] debug: unmapping init [mem 0xffffffffb8f14000-0xffffffffb8ffffff] [ 3.460927] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.468789] systemd[1]: Detected virtualization kvm. [ 3.471103] systemd[1]: Detected architecture x86-64. [ 3.473020] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.498551] systemd[1]: No hostname configured. [ 3.500129] systemd[1]: Set hostname to . [ 3.502119] random: systemd: uninitialized urandom read (16 bytes read) [ 3.504364] systemd[1]: Initializing machine ID from random generator. [ 3.547320] random: ln: uninitialized urandom read (6 bytes read) [ 3.619308] random: systemd: uninitialized urandom read (16 bytes read) [ 3.621740] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.626199] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.630421] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.128309] device-mapper: uevent: version 1.0.3 [ 4.129891] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 4.780062] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.787701] scsi host0: ata_piix [ 4.798184] random: fast init done [ 4.805310] scsi host1: ata_piix [ 4.809412] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.814242] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.547994] dracut-initqueue[577]: RTNETLINK answers: File exists [ 9.913222] random: crng init done [ 9.914386] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.211165] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.286452] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.514500] SELinux: Disabled at runtime. [ 11.579437] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.587980] systemd[1]: Detected virtualization kvm. [ 11.589465] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.057068] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.060249] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.065049] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.068842] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.072086] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.078326] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.082783] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ 12.234777] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.599992] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.878378] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.904206] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.059219] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.086267] EDAC sbridge: Ver: 1.1.2 [ 14.677960] Key type dns_resolver registered [ 14.990957] NFS: Registering the id_resolver key type [ 14.992553] Key type id_resolver registered [ 14.993845] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg655-server login: [ 40.542875] libcfs: loading out-of-tree module taints kernel. [ 40.562475] Key type ._llcrypt registered [ 40.564054] Key type .llcrypt registered [ 40.618816] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 48.842938] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 49.585495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.594540] alg: No test for adler32 (adler32-zlib) [ 50.626128] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 50.986142] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 52.615163] Key type lgssc registered [ 53.334842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.136094] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 110.883861] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 124.023580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 124.054587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 125.366893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 125.437168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 125.559831] Lustre: lustre-MDT0000: new disk, initializing [ 125.668230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 125.688215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 130.389641] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.412062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.533881] Lustre: 6479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 144.560437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 144.563936] Lustre: Skipped 1 previous similar message [ 144.666137] Lustre: lustre-MDT0001: new disk, initializing [ 144.748329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 144.777454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 144.787509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 149.519494] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.169223] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 163.549382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 163.761910] Lustre: lustre-OST0000: new disk, initializing [ 163.771619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 163.776849] Lustre: 8416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 163.858509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.925815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 164.942195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 165.019424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 170.358210] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.795311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.904305] Lustre: lustre-OST0001: new disk, initializing [ 184.910995] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 184.917632] Lustre: 9488:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.977828] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 191.719718] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.057837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 192.071814] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 192.125099] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 202.874763] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 210.764437] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 219.148594] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing check_logdir /tmp/testlogs/ [ 223.408804] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing yml_node [ 228.783334] Lustre: DEBUG MARKER: Client: 2.17.57.89 [ 230.762485] Lustre: DEBUG MARKER: MDS: 2.17.57.89 [ 233.142560] Lustre: DEBUG MARKER: OSS: 2.17.57.89 [ 234.501620] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 11:55:37 EDT 2026 [ 249.388026] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 250.698580] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 267.744663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 267.757818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 267.772148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 268.770992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.771898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 268.818516] Lustre: Skipped 3 previous similar messages [ 272.071119] Lustre: server umount lustre-MDT0000 complete [ 274.872012] hrtimer: interrupt took 6992671 ns [ 275.976475] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846180 with bad export cookie 3012699204614709840 [ 275.984108] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 275.994194] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 279.009567] LustreError: 6487:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.011073] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.024882] LustreError: 6487:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 6 previous similar messages [ 279.046588] Lustre: Skipped 2 previous similar messages [ 279.059189] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 279.074725] Lustre: Skipped 1 previous similar message [ 280.032469] LustreError: 6487:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.055983] LustreError: 6487:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 282.474814] Lustre: server umount lustre-MDT0001 complete [ 285.952207] Lustre: server umount lustre-OST0000 complete [ 290.057965] Lustre: server umount lustre-OST0001 complete [ 297.264048] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 307.197477] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 343.685498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 344.026337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 344.064695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 344.151459] Lustre: lustre-MDT0000: new disk, initializing [ 344.277740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 344.319979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 349.084196] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 358.911363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.012638] Lustre: 15038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 359.075815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 359.079071] Lustre: Skipped 1 previous similar message [ 359.182775] Lustre: lustre-MDT0001: new disk, initializing [ 359.307873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 359.359849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 359.374476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 363.683329] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 373.465111] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 380.195937] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 387.411477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 387.853975] Lustre: lustre-OST0000: new disk, initializing [ 387.859868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 387.877334] Lustre: 17010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 388.042758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.495876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 389.503130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 389.740482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 395.155546] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 403.503830] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 407.668380] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 407.887331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 412.321024] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 412.597594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 415.203737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.208278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 415.213175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 415.213183] Lustre: Skipped 2 previous similar messages [ 415.221709] Lustre: Skipped 2 previous similar messages [ 420.324834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 420.336612] Lustre: Skipped 2 previous similar messages [ 421.374181] Lustre: server umount lustre-OST0000 complete [ 425.443623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.455665] Lustre: Skipped 1 previous similar message [ 430.560047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 430.572186] Lustre: Skipped 3 previous similar messages [ 431.462461] Lustre: server umount lustre-MDT0000 complete [ 435.092534] LustreError: 15032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846339 with bad export cookie 3012699204614711373 [ 435.096991] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 435.116771] LustreError: 15032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 435.451727] Lustre: server umount lustre-MDT0001 complete [ 438.742548] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 11:59:01 (1787846341) [ 444.681258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.170761] LustreError: 19274:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.239976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.735604] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 450.534317] LustreError: 19275:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 455.648359] LustreError: 19274:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.246071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 461.448607] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 467.729929] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 472.311775] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 478.760853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 485.188441] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 492.097273] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 500.718485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.724860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.741032] Lustre: Skipped 1 previous similar message [ 500.769436] Lustre: Skipped 1 previous similar message [ 504.941042] Lustre: server umount lustre-OST0000 complete [ 508.897950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.711493] Lustre: server umount lustre-MDT0000 complete [ 516.064407] LustreError: 20338:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.093718] LustreError: 20338:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 518.275901] LustreError: 19256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846422 with bad export cookie 3012699204614712255 [ 518.278526] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.293014] LustreError: 19256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 518.695233] Lustre: server umount lustre-MDT0001 complete [ 531.663897] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 535.015057] Key type lgssc unregistered [ 535.369434] LNet: 22811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 535.383541] LNetError: 22811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 535.396742] LNet: Removed LNI 192.168.206.155@tcp [ 536.477211] Key type .llcrypt unregistered [ 536.480150] Key type ._llcrypt unregistered [ 560.158605] Key type ._llcrypt registered [ 560.162592] Key type .llcrypt registered [ 560.274696] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 561.684435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 561.714348] alg: No test for adler32 (adler32-zlib) [ 562.730632] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 563.075408] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 564.815190] Key type lgssc registered [ 565.906701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 573.902266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 573.932809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.429330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.857801] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 585.698346] LustreError: 24240:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.714135] LustreError: 24240:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 587.729514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.147818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 593.060394] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 600.750227] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 606.353431] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 613.094248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 613.568874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.168631] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 620.754500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 625.856383] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 641.509489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 641.510853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.525313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 641.532986] Lustre: Skipped 1 previous similar message [ 641.546388] Lustre: Skipped 1 previous similar message [ 643.418918] Lustre: server umount lustre-OST0000 complete [ 649.696409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 649.706403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.721804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 651.746686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.758977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 651.787259] Lustre: Skipped 1 previous similar message [ 653.176810] Lustre: server umount lustre-MDT0000 complete [ 656.680290] LustreError: 24219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846560 with bad export cookie 18002413562526370909 [ 656.690347] LustreError: 24219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 656.692846] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.905682] Lustre: server umount lustre-MDT0001 complete [ 666.234498] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 668.822351] Key type lgssc unregistered [ 669.136811] LNet: 28218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.150683] LNetError: 28218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 669.170614] LNet: Removed LNI 192.168.206.155@tcp [ 670.050493] Key type .llcrypt unregistered [ 670.053653] Key type ._llcrypt unregistered [ 693.567676] Key type ._llcrypt registered [ 693.571451] Key type .llcrypt registered [ 693.700282] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 694.793855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 694.807369] alg: No test for adler32 (adler32-zlib) [ 695.987959] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 696.317258] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 697.999177] Key type lgssc registered [ 699.359431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 707.665172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 707.689113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.425260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 714.086978] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 719.848146] LustreError: 29631:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 719.862340] LustreError: 29631:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 722.558424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.991175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 726.917431] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 734.294453] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 740.529362] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 753.517831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 753.857561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 754.877862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 760.387615] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 768.768527] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 775.652223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 775.675037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.702529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 777.696802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 777.701324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 777.716588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 777.759784] Lustre: Skipped 1 previous similar message [ 781.052901] Lustre: server umount lustre-OST0000 complete [ 787.936851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 787.951533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 787.956267] Lustre: Skipped 1 previous similar message [ 790.765315] Lustre: server umount lustre-MDT0000 complete [ 793.057101] LustreError: 30680:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 793.068786] LustreError: 30680:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 794.549570] LustreError: 29611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846698 with bad export cookie 13696783907525543815 [ 794.551703] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 794.560746] LustreError: 29611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 794.915902] Lustre: server umount lustre-MDT0001 complete [ 806.154526] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 808.796982] Key type lgssc unregistered [ 809.156035] LNet: 33511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.164564] LNetError: 33511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.202411] LNet: Removed LNI 192.168.206.155@tcp [ 810.237687] Key type .llcrypt unregistered [ 810.241577] Key type ._llcrypt unregistered [ 834.043449] Key type ._llcrypt registered [ 834.045331] Key type .llcrypt registered [ 834.161375] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 835.512968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 835.548243] alg: No test for adler32 (adler32-zlib) [ 836.531964] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 836.712895] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 838.399504] Key type lgssc registered [ 839.444638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 847.002493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 847.016704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.417857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 852.436829] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 858.620865] LustreError: 34922:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 858.656876] LustreError: 34922:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 859.985354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.640424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 865.235335] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 872.079565] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 877.386725] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 883.220649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 883.420386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 888.825564] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 893.675826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 895.804774] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 906.220380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 906.236394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.254649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 906.728094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 909.291455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 909.298652] Lustre: Skipped 2 previous similar messages [ 911.264285] Lustre: server umount lustre-OST0000 complete [ 916.960556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 916.967099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 916.980305] Lustre: Skipped 1 previous similar message [ 916.982487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 919.523159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.525620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 919.544650] Lustre: Skipped 1 previous similar message [ 920.937235] Lustre: server umount lustre-MDT0000 complete [ 924.064034] LustreError: 34905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846828 with bad export cookie 10278708999406064404 [ 924.074674] LustreError: 34905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 924.077776] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 924.323858] Lustre: server umount lustre-MDT0001 complete [ 939.747326] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 944.497079] Key type lgssc unregistered [ 944.906641] LNet: 38463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.922806] LNetError: 38463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.938798] LNet: Removed LNI 192.168.206.155@tcp [ 946.264530] Key type .llcrypt unregistered [ 946.266515] Key type ._llcrypt unregistered [ 970.401262] Key type ._llcrypt registered [ 970.405608] Key type .llcrypt registered [ 970.542549] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 971.882655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 972.021798] alg: No test for adler32 (adler32-zlib) [ 973.121140] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 973.364506] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 975.087237] Key type lgssc registered [ 976.324489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 984.594972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 984.638800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.021636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.989673] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 996.321890] LustreError: 39879:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.337935] LustreError: 39879:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 999.451600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.869712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1004.237355] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1011.693147] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1017.940439] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1024.810208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1025.274358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1031.421196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1032.050565] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1039.909523] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1050.086302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.095634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1050.099087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1050.111330] Lustre: Skipped 1 previous similar message [ 1052.278470] Lustre: server umount lustre-OST0000 complete [ 1075.686872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.699583] Lustre: Skipped 1 previous similar message [ 1075.707507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1075.715670] Lustre: Skipped 1 previous similar message [ 1078.261854] Lustre: server umount lustre-MDT0000 complete [ 1080.801234] LustreError: 39879:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1080.829138] LustreError: 39879:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 1082.223571] LustreError: 39860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846986 with bad export cookie 6067345403446548218 [ 1082.233334] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1088.741156] Lustre: server umount lustre-MDT0001 complete [ 1098.805831] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1102.311669] Key type lgssc unregistered [ 1102.615080] LNet: 43590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1102.625481] LNetError: 43590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1102.641708] LNet: Removed LNI 192.168.206.155@tcp [ 1103.906224] Key type .llcrypt unregistered [ 1103.908744] Key type ._llcrypt unregistered [ 1132.579990] Key type ._llcrypt registered [ 1132.588705] Key type .llcrypt registered [ 1132.700415] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1133.459833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1133.526360] alg: No test for adler32 (adler32-zlib) [ 1134.833657] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1135.268286] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1137.039153] Key type lgssc registered [ 1138.264526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1146.693799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1146.710083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.123624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1152.007673] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1158.624430] LustreError: 45003:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.645373] LustreError: 45003:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1159.535244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.804620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1164.173060] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1172.861490] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1179.624748] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1186.062616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1186.426265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.030417] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1193.611395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1198.824973] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1210.345120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.359781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1210.367158] Lustre: Skipped 2 previous similar messages [ 1210.405196] Lustre: Skipped 2 previous similar messages [ 1211.729815] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 1215.278322] Lustre: server umount lustre-MDT0000 complete [ 1215.488225] LustreError: 45004:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.515608] LustreError: 45004:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1216.829151] LustreError: 44998:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.691149] LustreError: 46847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847122 with bad export cookie 5890043839947863548 [ 1218.697333] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1218.712556] LustreError: 46847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1219.910631] LustreError: 46292:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.962414] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.55@tcp (stopping) [ 1220.582584] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.639283] LustreError: 45732:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000055fac653 x1874693496796416/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1223.585056] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1223.590141] Lustre: Skipped 1 previous similar message [ 1225.020191] LustreError: 45000:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1225.047540] LustreError: 45000:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1228.715828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1228.727039] Lustre: Skipped 1 previous similar message [ 1233.266799] LustreError: 44998:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.280192] LustreError: 44998:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1233.375803] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1233.779651] Lustre: server umount lustre-MDT0001 complete [ 1248.670759] Lustre: server umount lustre-OST0000 complete [ 1266.719488] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1270.009935] Key type lgssc unregistered [ 1270.330635] LNet: 48868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.342339] LNetError: 48868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.368957] LNet: Removed LNI 192.168.206.155@tcp [ 1271.248445] Key type .llcrypt unregistered [ 1271.250453] Key type ._llcrypt unregistered [ 1295.730390] Key type ._llcrypt registered [ 1295.733769] Key type .llcrypt registered [ 1295.822220] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1296.938249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1296.966813] alg: No test for adler32 (adler32-zlib) [ 1297.980315] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1298.133464] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1299.759314] Key type lgssc registered [ 1300.850864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1307.425976] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1307.440287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1322.975813] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1328.098494] LustreError: 50255:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [ 1354.783673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1360.181635] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1366.964259] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1535.732614] Lustre: server umount lustre-OST0000 complete [ 1553.019490] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1555.608968] Key type lgssc unregistered [ 1555.894731] LNet: 51932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.902274] LNetError: 51932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.924058] LNet: Removed LNI 192.168.206.155@tcp [ 1556.772601] Key type .llcrypt unregistered [ 1556.777204] Key type ._llcrypt unregistered [ 1582.229557] Key type ._llcrypt registered [ 1582.232838] Key type .llcrypt registered [ 1582.384203] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1583.392912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1583.564213] alg: No test for adler32 (adler32-zlib) [ 1584.754267] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1585.002664] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1586.687161] Key type lgssc registered [ 1587.832211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1596.102624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1596.139220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1597.607802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1602.402114] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1607.150863] LustreError: 53347:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.172195] LustreError: 53347:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1611.384394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.881907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1616.536750] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1624.945505] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1631.114174] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1638.522910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1638.985375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.110787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1645.568890] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1652.968771] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1656.161545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.161829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.188444] Lustre: Skipped 1 previous similar message [ 1661.410516] Lustre: server umount lustre-OST0000 complete [ 1666.531229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.535472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1666.553988] Lustre: Skipped 1 previous similar message [ 1666.573086] Lustre: Skipped 1 previous similar message [ 1671.649433] LustreError: 53344:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1671.664209] LustreError: 53344:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 1671.728761] Lustre: server umount lustre-MDT0000 complete [ 1675.312678] LustreError: 53329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847579 with bad export cookie 13909706480872847252 [ 1675.328166] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1675.540767] Lustre: server umount lustre-MDT0001 complete [ 1686.557828] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1689.367779] Key type lgssc unregistered [ 1689.614596] LNet: 56882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.620512] LNetError: 56882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.643116] LNet: Removed LNI 192.168.206.155@tcp [ 1690.627383] Key type .llcrypt unregistered [ 1690.629210] Key type ._llcrypt unregistered [ 1715.862747] Key type ._llcrypt registered [ 1715.864640] Key type .llcrypt registered [ 1716.043665] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1717.569312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1717.647246] alg: No test for adler32 (adler32-zlib) [ 1718.801837] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1719.028821] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1720.703183] Key type lgssc registered [ 1722.223181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1731.229527] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1731.252304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1746.975443] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1752.095318] LustreError: 58270:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [ 1778.784089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.088250] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1793.856940] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1801.470746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.111449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.213038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1806.558963] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1807.330864] LustreError: 58296:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.360738] LustreError: 58296:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 1812.471568] LustreError: 58295:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1813.491774] LustreError: 59307:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1813.529894] LustreError: 59307:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 1814.482698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.862809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1819.019629] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1826.475226] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1832.511426] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1835.493608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1835.499969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.507562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1835.536825] Lustre: Skipped 1 previous similar message [ 1835.565415] Lustre: Skipped 1 previous similar message [ 1840.609339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1840.624368] Lustre: Skipped 1 previous similar message [ 1840.818368] Lustre: server umount lustre-OST0000 complete [ 1865.825158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.828429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1865.837286] Lustre: Skipped 1 previous similar message [ 1868.038920] Lustre: server umount lustre-MDT0000 complete [ 1871.334328] LustreError: 59310:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.359023] LustreError: 59310:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 1871.830758] LustreError: 60050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847775 with bad export cookie 2888855489359296112 [ 1871.840902] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.844830] LustreError: 60050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1886.175119] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1886.428635] Lustre: server umount lustre-MDT0001 complete [ 1896.468805] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1899.493600] Key type lgssc unregistered [ 1899.858340] LNet: 62018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1899.877454] LNetError: 62018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1899.893248] LNet: Removed LNI 192.168.206.155@tcp [ 1900.721608] Key type .llcrypt unregistered [ 1900.723465] Key type ._llcrypt unregistered [ 1924.064807] Key type ._llcrypt registered [ 1924.069696] Key type .llcrypt registered [ 1924.197895] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1925.392060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1925.637031] alg: No test for adler32 (adler32-zlib) [ 1926.734202] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1927.026500] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1928.775542] Key type lgssc registered [ 1929.798797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1937.245722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1937.258664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.668105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.422424] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1948.130212] LustreError: 63436:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1948.143721] LustreError: 63436:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1949.371631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.759548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1953.812869] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1960.619942] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1965.909610] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1972.618742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1972.909373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.197213] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1983.162150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1984.608883] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2003.262623] Lustre: server umount lustre-OST0000 complete [ 2009.058781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.077220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2009.080272] Lustre: Skipped 1 previous similar message [ 2014.185143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.189891] Lustre: Skipped 2 previous similar messages [ 2017.325133] Lustre: server umount lustre-MDT0000 complete [ 2019.297073] LustreError: 63436:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.314953] LustreError: 63436:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 2020.256100] Lustre: 62749:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787847908/real 1787847908] req@000000004ba551c2 x1874694327267328/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787847924 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2020.289144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.320794] LustreError: 64165:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.347748] LustreError: 64165:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 2020.447968] LustreError: 63416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847924 with bad export cookie 14761228238513659075 [ 2020.459537] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2020.470712] LustreError: 63416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2020.788766] Lustre: server umount lustre-MDT0001 complete [ 2031.449947] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2034.346364] Key type lgssc unregistered [ 2034.762868] LNet: 66993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2034.786054] LNetError: 66993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2034.812750] LNet: Removed LNI 192.168.206.155@tcp [ 2035.860581] Key type .llcrypt unregistered [ 2035.862911] Key type ._llcrypt unregistered [ 2061.357746] Key type ._llcrypt registered [ 2061.360632] Key type .llcrypt registered [ 2061.433409] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2062.023823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.036737] alg: No test for adler32 (adler32-zlib) [ 2063.217308] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2063.525677] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2065.223386] Key type lgssc registered [ 2066.843793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2077.664410] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 12:26:20 (1787847980) [ 2085.748276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2085.765614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.260972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.538720] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2097.641910] LustreError: 68699:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.667330] LustreError: 68699:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2099.912711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.275467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2104.276822] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2110.744818] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2117.024448] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2124.694841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2125.042060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.864863] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2132.277913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2139.023919] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2145.124375] Lustre: Failing over lustre-MDT0000 [ 2145.507402] Lustre: server umount lustre-MDT0000 complete [ 2146.275111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2146.294417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.307971] LustreError: 68700:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.816673] LustreError: 68699:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.818069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.846345] LustreError: 68699:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 2147.871667] Lustre: Skipped 1 previous similar message [ 2151.739409] LustreError: 68695:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.781162] LustreError: 68695:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2156.808885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.862967] LustreError: 68694:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2156.874916] LustreError: 68694:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 2156.916770] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2157.047749] Lustre: *** cfs_fail_loc=135, val=0*** [ 2157.125995] LustreError: 71784:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2157.129157] LustreError: 71784:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.155@tcp: cfg command failed: rc = -2 [ 2157.135957] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2157.135957] [ 2157.146242] LustreError: MGC192.168.206.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2157.154899] LustreError: 71773:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2157.160699] LustreError: 71773:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2157.167263] LustreError: 71773:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2157.170774] LustreError: 71773:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2157.183488] Lustre: server umount lustre-MDT0000 complete [ 2157.186653] LustreError: 71773:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2165.023574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2165.265507] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2165.282887] LustreError: 68692:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2165.704760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.773574] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2168.127908] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2170.866747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2170.907061] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2170.935869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2171.736483] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2181.096087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.101268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2181.109990] Lustre: Skipped 1 previous similar message [ 2186.223494] LustreError: 70522:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.240658] LustreError: 70522:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 8 previous similar messages [ 2186.327720] Lustre: server umount lustre-OST0000 complete [ 2191.328442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.329243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2191.344639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2191.362731] Lustre: Skipped 2 previous similar messages [ 2196.460300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2196.463814] Lustre: Skipped 1 previous similar message [ 2196.979743] Lustre: server umount lustre-MDT0000 complete [ 2201.509884] LustreError: 68681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848105 with bad export cookie 15534101219767211261 [ 2201.520403] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2201.521396] LustreError: 68681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2202.012722] Lustre: server umount lustre-MDT0001 complete [ 2213.995149] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2217.127866] Key type lgssc unregistered [ 2217.462810] LNet: 73566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.468908] LNetError: 73566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.485745] LNet: Removed LNI 192.168.206.155@tcp [ 2218.513409] Key type .llcrypt unregistered [ 2218.515534] Key type ._llcrypt unregistered [ 2241.416760] Key type ._llcrypt registered [ 2241.420717] Key type .llcrypt registered [ 2241.571514] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2242.871183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2242.989770] alg: No test for adler32 (adler32-zlib) [ 2244.093752] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2244.417382] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2246.103216] Key type lgssc registered [ 2247.411262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2255.396706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2255.423962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.771657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.627556] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2266.081946] LustreError: 74981:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.108701] LustreError: 74981:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2270.738717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.190524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2277.735741] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2285.711647] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2291.337740] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2297.811278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2298.115964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2301.235878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2304.947801] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2313.160347] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2323.190180] Lustre: Failing over lustre-MDT0000 [ 2323.467195] Lustre: server umount lustre-MDT0000 complete [ 2326.372735] LustreError: 74976:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.011342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.041922] Lustre: Skipped 2 previous similar messages [ 2331.464047] LustreError: 74975:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.496896] LustreError: 74975:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 2333.234962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2333.419501] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2333.568613] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2333.787432] LustreError: 78067:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2333.796124] LustreError: 78067:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.155@tcp: cfg command failed: rc = -9 [ 2333.810134] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2333.810134] [ 2333.823606] LustreError: MGC192.168.206.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2333.850222] LustreError: 78056:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2333.859256] LustreError: 78056:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2333.866334] LustreError: 78056:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2333.875389] LustreError: 78056:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2333.928115] Lustre: server umount lustre-MDT0000 complete [ 2333.930970] LustreError: 78056:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2336.572624] LustreError: 74977:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2336.603348] LustreError: 74977:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 2341.221397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.476559] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.784933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.827420] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2346.223759] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2346.824569] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2346.989299] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2347.030644] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2347.095717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2357.222338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2357.225450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2357.238541] Lustre: Skipped 1 previous similar message [ 2357.256726] Lustre: Skipped 1 previous similar message [ 2359.914613] Lustre: server umount lustre-OST0000 complete [ 2367.459839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.477629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2367.479492] Lustre: Skipped 1 previous similar message [ 2367.480889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2367.498635] Lustre: Skipped 1 previous similar message [ 2370.177697] Lustre: server umount lustre-MDT0000 complete [ 2372.579388] LustreError: 74980:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.607503] LustreError: 74980:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 4 previous similar messages [ 2374.404234] LustreError: 74962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848278 with bad export cookie 14590712549214285993 [ 2374.423711] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.872117] Lustre: server umount lustre-MDT0001 complete [ 2385.354736] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2388.556510] Key type lgssc unregistered [ 2388.860769] LNet: 79851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2388.881555] LNetError: 79851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.918378] LNet: Removed LNI 192.168.206.155@tcp [ 2389.881877] Key type .llcrypt unregistered [ 2389.886181] Key type ._llcrypt unregistered [ 2401.855423] Key type ._llcrypt registered [ 2401.857844] Key type .llcrypt registered [ 2401.940385] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2402.201268] Key type .llcrypt unregistered [ 2402.203908] Key type ._llcrypt unregistered [ 2424.640188] Key type ._llcrypt registered [ 2424.642718] Key type .llcrypt registered [ 2424.762110] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2425.845624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2425.973354] alg: No test for adler32 (adler32-zlib) [ 2427.090222] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2427.330243] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2429.015207] Key type lgssc registered [ 2430.308877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2438.094642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2438.120392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.678652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2444.381241] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2449.897079] LustreError: 81760:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.920869] LustreError: 81760:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2453.221854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.710208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2458.788986] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2466.403676] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2471.502732] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2477.819655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2478.171690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.349684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2484.557766] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2491.944440] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2505.188041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2505.191177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.200467] LustreError: Skipped 1 previous similar message [ 2505.203420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2505.228517] Lustre: Skipped 1 previous similar message [ 2507.856179] Lustre: server umount lustre-OST0000 complete [ 2515.431574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.433839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2515.449906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2515.449915] Lustre: Skipped 1 previous similar message [ 2515.452039] Lustre: Skipped 1 previous similar message [ 2517.795428] Lustre: server umount lustre-MDT0000 complete [ 2520.547438] LustreError: 82487:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.581499] LustreError: 82487:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 2522.531600] LustreError: 81740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848426 with bad export cookie 5559564206345419894 [ 2522.542251] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2522.544713] LustreError: 81740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2522.828030] Lustre: server umount lustre-MDT0001 complete [ 2533.731557] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2536.321894] Key type lgssc unregistered [ 2536.624739] LNet: 85294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2536.633154] LNetError: 85294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2536.660598] LNet: Removed LNI 192.168.206.155@tcp [ 2537.521206] Key type .llcrypt unregistered [ 2537.526361] Key type ._llcrypt unregistered [ 2560.529843] Key type ._llcrypt registered [ 2560.535187] Key type .llcrypt registered [ 2560.668985] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2561.807229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2561.844688] alg: No test for adler32 (adler32-zlib) [ 2562.974527] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2563.173418] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2564.887157] Key type lgssc registered [ 2566.015145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2573.187262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2573.204352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.575856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.037371] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2584.039830] LustreError: 86722:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.057469] LustreError: 86722:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2588.224404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.842914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2593.519382] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2601.091182] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2606.476932] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2613.179455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2613.466602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2619.458186] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2619.653090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2626.419844] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2634.989510] Lustre: server umount lustre-OST0000 complete [ 2639.840630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2639.857984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.875634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2645.473858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2645.479543] Lustre: Skipped 3 previous similar messages [ 2648.607461] Lustre: server umount lustre-MDT0000 complete [ 2650.598147] LustreError: 87450:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.495852] Lustre: 86034:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787848539/real 1787848539] req@000000001234f16c x1874694994157696/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787848555 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2651.536595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.575880] Lustre: Skipped 1 previous similar message [ 2652.234905] LustreError: 89300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848556 with bad export cookie 12908817456245563970 [ 2652.242964] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2652.249514] LustreError: 89300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2652.551317] Lustre: server umount lustre-MDT0001 complete [ 2662.692232] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2665.343735] Key type lgssc unregistered [ 2665.641514] LNet: 90256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2665.651704] LNetError: 90256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2665.668235] LNet: Removed LNI 192.168.206.155@tcp [ 2666.497300] Key type .llcrypt unregistered [ 2666.507270] Key type ._llcrypt unregistered [ 2688.689756] Key type ._llcrypt registered [ 2688.691405] Key type .llcrypt registered [ 2688.788188] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2690.108439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2690.153114] alg: No test for adler32 (adler32-zlib) [ 2691.250045] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2691.603527] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2693.375308] Key type lgssc registered [ 2694.663453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2702.942957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2702.964040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.331298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2708.641506] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2714.597825] LustreError: 91664:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.625297] LustreError: 91664:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2717.032876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2717.465790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2722.515949] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2729.737644] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2734.987504] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2741.106744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2741.427097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2747.854419] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2749.624950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2754.933753] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2766.834554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2766.846576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.880160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2768.353370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2768.360422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.377824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2770.430932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.881307] Lustre: server umount lustre-OST0000 complete [ 2775.521780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.543495] Lustre: Skipped 1 previous similar message [ 2775.547398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.562446] Lustre: Skipped 2 previous similar messages [ 2780.640745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2780.657309] Lustre: Skipped 1 previous similar message [ 2781.718956] Lustre: server umount lustre-MDT0000 complete [ 2785.265628] LustreError: 91644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848689 with bad export cookie 1601720376127731751 [ 2785.271042] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2785.280293] LustreError: 91644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2785.630218] Lustre: server umount lustre-MDT0001 complete [ 2795.431699] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2798.400092] Key type lgssc unregistered [ 2798.734555] LNet: 95200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2798.738644] LNetError: 95200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2798.750877] LNet: Removed LNI 192.168.206.155@tcp [ 2799.622573] Key type .llcrypt unregistered [ 2799.626503] Key type ._llcrypt unregistered [ 2822.498909] Key type ._llcrypt registered [ 2822.502985] Key type .llcrypt registered [ 2822.611260] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2823.806255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2823.900108] alg: No test for adler32 (adler32-zlib) [ 2824.999652] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2825.230246] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2826.887119] Key type lgssc registered [ 2828.032882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2836.155443] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2836.177597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.872695] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2856.991307] LustreError: 96589:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [ 2883.685653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.552484] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2896.986752] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2902.554551] Lustre: server umount lustre-OST0000 complete [ 2910.242066] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 12:40:13 (1787848813) [ 2916.959357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.464042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.051778] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2922.472997] LustreError: 98227:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2922.494394] LustreError: 98227:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2927.606131] LustreError: 98228:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2930.475918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2930.813505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2935.042660] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2941.245528] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2947.381554] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2954.409341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2954.593567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.214633] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2962.846178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2970.050809] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2981.344845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2981.350234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.363635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2983.399365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2983.404463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2983.404649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2987.415747] Lustre: server umount lustre-OST0000 complete [ 2992.098244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2992.106584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.132174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2992.134762] Lustre: Skipped 1 previous similar message [ 2996.872479] Lustre: server umount lustre-MDT0000 complete [ 2998.755755] LustreError: 98228:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.787453] LustreError: 98228:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 3000.498381] LustreError: 98209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848904 with bad export cookie 14541383442300565856 [ 3000.507899] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3000.516563] LustreError: 98209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3000.794243] Lustre: server umount lustre-MDT0001 complete [ 3013.174787] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3015.948630] Key type lgssc unregistered [ 3016.200189] LNet: 101924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.205614] LNetError: 101924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3016.221427] LNet: Removed LNI 192.168.206.155@tcp [ 3017.165394] Key type .llcrypt unregistered [ 3017.171114] Key type ._llcrypt unregistered [ 3059.842262] Key type ._llcrypt registered [ 3059.843896] Key type .llcrypt registered [ 3060.054791] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3061.315608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3061.369109] alg: No test for adler32 (adler32-zlib) [ 3062.579237] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 3062.898459] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3064.585607] Key type lgssc registered [ 3065.764666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3073.289838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3073.313140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.865078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3079.598144] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3084.262854] LustreError: 104111:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3084.297862] LustreError: 104111:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 3087.710439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.245699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3093.057628] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3100.641208] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3106.357324] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3113.075441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.450547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.614480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3119.859709] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3126.934742] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3140.064968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3140.066271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.073373] LustreError: Skipped 1 previous similar message [ 3140.076443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.083380] Lustre: Skipped 1 previous similar message [ 3140.100028] Lustre: Skipped 1 previous similar message [ 3145.186234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3145.194774] Lustre: Skipped 1 previous similar message [ 3150.311260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3150.327437] Lustre: Skipped 1 previous similar message [ 3151.327228] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3151.502777] Lustre: server umount lustre-OST0000 complete [ 3155.424957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.431375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3155.446911] Lustre: Skipped 1 previous similar message [ 3160.547653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3160.556424] Lustre: Skipped 2 previous similar messages [ 3161.407769] Lustre: server umount lustre-MDT0000 complete [ 3164.725295] LustreError: 104093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849068 with bad export cookie 9365307170692289343 [ 3164.734304] LustreError: 104093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3164.738443] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3165.065678] Lustre: server umount lustre-MDT0001 complete [ 3176.245655] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3179.368640] Key type lgssc unregistered [ 3179.689765] LNet: 107658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3179.706426] LNetError: 107658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3179.744493] LNet: Removed LNI 192.168.206.155@tcp [ 3180.877307] Key type .llcrypt unregistered [ 3180.883719] Key type ._llcrypt unregistered [ 3201.041326] Key type ._llcrypt registered [ 3201.043866] Key type .llcrypt registered [ 3201.133348] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3202.115303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3202.133709] alg: No test for adler32 (adler32-zlib) [ 3203.263402] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 3203.540928] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3205.271205] Key type lgssc registered [ 3206.541750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3215.290770] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3215.338023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3230.943697] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3236.063616] LustreError: 108902:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [ 3262.688060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3269.220236] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3277.463462] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3284.938757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.189579] LustreError: MGC192.168.206.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3285.199298] LustreError: 109925:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3285.206554] LustreError: 109925:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3285.218971] LustreError: 109925:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3285.345978] Lustre: server umount lustre-MDT0000 complete [ 3285.352666] LustreError: 109925:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3306.463768] Lustre: 108239:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849194/real 1787849194] req@000000001057c6c6 x1874695665224960/t0(0) o400->MGC192.168.206.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787849210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3306.508954] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3306.523464] LustreError: 108915:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3327.092680] Lustre: server umount lustre-OST0000 complete [ 3335.466850] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 3343.035718] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3381.868247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.116758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3382.149053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3382.253258] Lustre: lustre-MDT0000: new disk, initializing [ 3382.364971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.397111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3386.910209] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3397.300679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.417478] Lustre: 112723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3397.495892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3397.502019] Lustre: Skipped 1 previous similar message [ 3397.581941] Lustre: lustre-MDT0001: new disk, initializing [ 3397.676378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3397.753615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3397.769326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3401.870429] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3410.302081] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3415.442749] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3421.334266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3421.615987] Lustre: lustre-OST0000: new disk, initializing [ 3421.622949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3421.640770] Lustre: 114688:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3421.743632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3427.415307] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3427.904450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3427.915029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3428.016109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3435.722390] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3440.208157] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3440.445498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3444.821493] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3445.067319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3448.805728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3448.809267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.815301] LustreError: Skipped 1 previous similar message [ 3448.818186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3448.856950] Lustre: Skipped 1 previous similar message [ 3453.559352] Lustre: server umount lustre-OST0000 complete [ 3459.045022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.047203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3459.063077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3459.068156] Lustre: Skipped 2 previous similar messages [ 3464.160802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3464.163525] Lustre: Skipped 1 previous similar message [ 3469.283253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3469.302904] Lustre: Skipped 1 previous similar message [ 3471.327190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3471.468102] Lustre: server umount lustre-MDT0000 complete [ 3475.501369] LustreError: 112715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849379 with bad export cookie 10985480584394392287 [ 3475.505754] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3475.518397] LustreError: 112715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3475.834053] Lustre: server umount lustre-MDT0001 complete [ 3484.210675] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 12:49:47 (1787849387) [ 3486.993612] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3501.218591] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 3510.088320] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3544.018534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.302224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3544.329770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3544.422656] Lustre: lustre-MDT0000: new disk, initializing [ 3544.514889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3544.549919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3549.123468] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3560.556434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3560.650537] Lustre: 119591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3560.692457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3560.696364] Lustre: Skipped 1 previous similar message [ 3560.772461] Lustre: lustre-MDT0001: new disk, initializing [ 3560.852760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3560.894837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3560.913358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3565.963170] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3575.022176] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3580.758196] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3587.029962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3587.277474] Lustre: lustre-OST0000: new disk, initializing [ 3587.281931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3587.287665] Lustre: 121558:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3587.355915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.305897] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3594.278578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3594.283824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3594.386529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3602.632146] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3609.330964] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3609.529415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3616.531573] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3616.723885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3624.929049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3624.943919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.970261] Lustre: Skipped 1 previous similar message [ 3624.983300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3626.746212] Lustre: server umount lustre-OST0000 complete [ 3632.608256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3632.628212] LustreError: Skipped 1 previous similar message [ 3632.644126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.661252] Lustre: Skipped 1 previous similar message [ 3635.685462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3635.698025] Lustre: Skipped 4 previous similar messages [ 3637.792425] Lustre: server umount lustre-MDT0000 complete [ 3640.809204] LustreError: 119604:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3640.827515] LustreError: 119604:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 4 previous similar messages [ 3641.642447] LustreError: 119585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849545 with bad export cookie 10985480584394393169 [ 3641.645866] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3641.651964] LustreError: 119585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3642.308649] Lustre: server umount lustre-MDT0001 complete [ 3650.646813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3651.192058] LustreError: 123774:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3651.296160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3656.408464] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3656.682773] LustreError: 123775:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.793129] LustreError: 123774:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.661097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3670.553880] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3677.045490] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3683.001986] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3689.116645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3695.938862] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3703.600680] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3710.010225] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3717.601226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.608244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3717.614316] Lustre: Skipped 2 previous similar messages [ 3717.635450] Lustre: Skipped 1 previous similar message [ 3720.069698] Lustre: server umount lustre-OST0000 complete [ 3727.328710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3727.342579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.265730] Lustre: server umount lustre-MDT0000 complete [ 3732.960618] LustreError: 123775:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3732.984855] LustreError: 123775:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 3733.764020] LustreError: 123754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849637 with bad export cookie 10985480584394394051 [ 3733.773929] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3733.779153] LustreError: 123754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3734.230925] Lustre: server umount lustre-MDT0001 complete [ 3744.868782] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3747.429928] Key type lgssc unregistered [ 3747.632293] LNet: 127408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3747.637056] LNetError: 127408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3747.653962] LNet: Removed LNI 192.168.206.155@tcp [ 3748.413676] Key type .llcrypt unregistered [ 3748.419808] Key type ._llcrypt unregistered [ 3763.888907] Key type ._llcrypt registered [ 3763.892425] Key type .llcrypt registered [ 3763.992216] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 3778.364305] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3779.122444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3779.218724] alg: No test for adler32 (adler32-zlib) [ 3780.244253] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 3780.516834] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3782.159188] Key type lgssc registered [ 3783.471270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3814.446277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3814.499279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.788031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3815.848116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3815.930100] Lustre: lustre-MDT0000: new disk, initializing [ 3816.017020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3816.033695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3820.336216] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3830.232226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3830.365901] Lustre: 130565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3830.437242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3830.445464] Lustre: Skipped 1 previous similar message [ 3830.537506] Lustre: lustre-MDT0001: new disk, initializing [ 3830.639027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3830.673705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3830.681703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3834.795557] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3844.234612] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3849.933531] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3856.600026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3856.850568] Lustre: lustre-OST0000: new disk, initializing [ 3856.857963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3856.865546] Lustre: 132533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3856.978955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3863.612393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3863.634095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3863.751496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3863.807066] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3873.717785] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3878.911939] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3879.137378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3884.010225] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3884.244689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3889.649878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3889.653510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3889.663775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3889.687048] LustreError: Skipped 1 previous similar message [ 3894.768267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3894.781533] Lustre: Skipped 2 previous similar messages [ 3899.880508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3899.890409] Lustre: Skipped 1 previous similar message [ 3900.383169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3900.586683] Lustre: server umount lustre-OST0000 complete [ 3904.996835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.000320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3905.017229] Lustre: Skipped 2 previous similar messages [ 3905.031711] Lustre: Skipped 1 previous similar message [ 3910.113707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3910.116386] Lustre: Skipped 1 previous similar message [ 3910.367655] Lustre: server umount lustre-MDT0000 complete [ 3913.726349] LustreError: 130556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849817 with bad export cookie 11473453253144020906 [ 3913.736195] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3914.021161] Lustre: server umount lustre-MDT0001 complete [ 3922.020284] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 12:57:05 (1787849825) [ 3928.322330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.802043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3932.786962] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3934.186047] LustreError: 134945:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3934.214492] LustreError: 134945:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 3938.278611] LustreError: 134946:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3940.838255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3941.231634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3945.785335] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3952.469155] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3956.438852] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3958.753039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3958.760413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3958.772495] Lustre: Skipped 1 previous similar message [ 3958.786884] Lustre: Skipped 1 previous similar message [ 3964.348633] Lustre: server umount lustre-MDT0000 complete [ 3965.926936] LustreError: 134945:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3965.951763] LustreError: 134945:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 3967.456966] LustreError: 134926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849871 with bad export cookie 11473453253144021788 [ 3967.463686] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3967.470773] LustreError: 134926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3967.845467] Lustre: server umount lustre-MDT0001 complete [ 3974.588558] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 12:57:57 (1787849877) [ 3980.397574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3995.999578] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4001.119441] LustreError: 137357:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [ 4026.847277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4033.175623] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4040.210979] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4042.035483] Lustre: server umount lustre-OST0000 complete [ 4048.860095] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 12:59:12 (1787849952) [ 4055.005993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4055.460972] LustreError: 138803:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4055.515606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.730728] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4060.647219] LustreError: 138804:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.409682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.176576] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4079.704812] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4084.728278] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4091.577445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4091.953505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.960051] Lustre: Skipped 1 previous similar message [ 4099.181348] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4106.706301] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4114.915275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4114.928565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.942843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4114.950611] Lustre: Skipped 2 previous similar messages [ 4119.572328] Lustre: server umount lustre-MDT0000 complete [ 4121.570402] LustreError: 138804:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4121.595217] LustreError: 138804:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 4 previous similar messages [ 4123.035199] LustreError: 138784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850027 with bad export cookie 11473453253144022453 [ 4123.038759] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4123.049615] LustreError: 138784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4123.314472] Lustre: server umount lustre-MDT0001 complete [ 4136.795986] Lustre: server umount lustre-OST0000 complete [ 4143.576627] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 13:00:46 (1787850046) [ 4149.026056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4149.466134] LustreError: 142455:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4153.209879] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4161.401241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.809252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4161.815100] Lustre: Skipped 1 previous similar message [ 4166.557950] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4174.513178] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4180.025221] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4187.230143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.953889] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4194.874779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4200.602900] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4204.581820] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4204.811474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4208.810383] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4209.022134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4211.177871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.182055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4211.192965] Lustre: Skipped 2 previous similar messages [ 4211.196488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4211.213091] Lustre: Skipped 4 previous similar messages [ 4213.096940] Lustre: server umount lustre-OST0000 complete [ 4218.337238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4221.411323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.431976] Lustre: Skipped 2 previous similar messages [ 4223.114933] Lustre: server umount lustre-MDT0000 complete [ 4226.528519] LustreError: 143472:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4226.548420] LustreError: 143472:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 4226.737904] LustreError: 142437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850130 with bad export cookie 11473453253144023517 [ 4226.742890] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4226.753926] LustreError: 142437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4227.154776] Lustre: server umount lustre-MDT0001 complete [ 4234.492456] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 13:02:17 (1787850137) [ 4240.551602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4256.159586] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4261.279483] LustreError: 146623:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [ 4292.473690] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4297.998570] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4303.619466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.940537] LustreError: 146648:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4303.992281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.996469] Lustre: Skipped 2 previous similar messages [ 4304.037404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4308.337748] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4317.252538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4321.835162] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4329.200991] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4335.709207] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4340.985926] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4341.228323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4345.364403] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4345.531043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4347.364983] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.371428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4347.407046] Lustre: Skipped 1 previous similar message [ 4347.414754] Lustre: Skipped 3 previous similar messages [ 4353.314510] Lustre: server umount lustre-MDT0000 complete [ 4356.609791] LustreError: 146629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850260 with bad export cookie 11473453253144024546 [ 4356.631231] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4356.979682] Lustre: server umount lustre-MDT0001 complete [ 4370.788915] Lustre: server umount lustre-OST0000 complete [ 4377.973368] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 13:04:41 (1787850281) [ 4384.191157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4399.776930] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4404.895430] LustreError: 150818:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [ 4436.091589] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4443.791731] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4449.754435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.320888] LustreError: 150843:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.346306] LustreError: 150843:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 4450.561057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4454.493388] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4462.038037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4465.936597] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4472.839377] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4477.698981] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4483.476137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4483.643504] Lustre: lustre-OST0001: new disk, initializing [ 4483.646503] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4483.651283] Lustre: 153691:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4489.240969] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4493.375384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4493.389839] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4493.461200] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4497.515812] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4501.469941] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4501.654745] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4506.005545] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4506.172566] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4508.640955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4508.649221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4508.662799] Lustre: Skipped 1 previous similar message [ 4508.669044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4508.675680] Lustre: Skipped 5 previous similar messages [ 4513.810343] Lustre: server umount lustre-OST0000 complete [ 4519.399588] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4519.415070] LustreError: Skipped 1 previous similar message [ 4523.502466] Lustre: server umount lustre-OST0001 complete [ 4533.338931] Lustre: server umount lustre-MDT0000 complete [ 4536.335336] LustreError: 150824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850440 with bad export cookie 11473453253144025498 [ 4536.337974] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4536.344315] LustreError: 150824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4536.761665] Lustre: server umount lustre-MDT0001 complete [ 4555.750631] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 13:07:38 (1787850458) [ 4557.311156] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4558.881551] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4560.363447] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 13:07:43 (1787850463) [ 4566.605967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.800766] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4566.850133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4567.027273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.032798] Lustre: Skipped 5 previous similar messages [ 4571.464916] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4578.394304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.468655] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4578.519315] Lustre: 157247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4582.227332] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4589.066521] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4594.104598] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4600.079806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4600.277101] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4606.592744] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4608.150968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4613.825688] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4618.405857] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4618.606376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4623.580816] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4623.814400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4628.449654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4628.457522] LustreError: Skipped 1 previous similar message [ 4628.462804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4628.490362] Lustre: Skipped 5 previous similar messages [ 4640.223120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4640.363701] Lustre: server umount lustre-OST0000 complete [ 4663.912559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4669.479523] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4669.493488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4669.493636] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4669.525641] Lustre: Skipped 1 previous similar message [ 4670.699241] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4677.494758] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4681.434794] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4681.556790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4685.041321] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4685.210377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4690.057040] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9454d8f7e000.ost_server_uuid 50 [ 4692.005426] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9454d8f7e000.ost_server_uuid in FULL state after 0 sec [ 4692.200537] LustreError: 157255:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.55@tcp) returned error from blocking AST (req@00000000047cde46 x1874696270503680 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000014ca191d/0x9f39e4f1dcb322f4 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.55@tcp remote: 0xfe497fd6affcc2b7 expref: 5 pid: 157254 timeout: 4792 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4692.244732] LustreError: lustre-MDT0000: A client on nid 192.168.206.55@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4692.259342] LustreError: 157243:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.55@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000014ca191d/0x9f39e4f1dcb322f4 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.55@tcp remote: 0xfe497fd6affcc2b7 expref: 6 pid: 157254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4698.086321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4698.096581] LustreError: Skipped 1 previous similar message [ 4702.942066] Lustre: server umount lustre-OST0000 complete [ 4712.904269] Lustre: server umount lustre-MDT0000 complete [ 4715.491497] LustreError: 157259:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4715.511892] LustreError: 157259:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 9 previous similar messages [ 4715.978225] LustreError: 157240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850620 with bad export cookie 11473453253144026681 [ 4715.988448] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4715.990073] LustreError: 157240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4730.335119] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4730.632106] Lustre: server umount lustre-MDT0001 complete [ 4741.637030] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 4744.434953] Key type lgssc unregistered [ 4744.747681] LNet: 163362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4744.753420] LNetError: 163362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4744.772641] LNet: Removed LNI 192.168.206.155@tcp [ 4745.650053] Key type .llcrypt unregistered [ 4745.652845] Key type ._llcrypt unregistered [ 4768.842457] Key type ._llcrypt registered [ 4768.851208] Key type .llcrypt registered [ 4769.004793] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4770.447757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4770.470081] alg: No test for adler32 (adler32-zlib) [ 4771.562672] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 4771.860874] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 4773.591151] Key type lgssc registered [ 4774.851559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4782.445794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4782.486617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4783.937502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.166395] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4793.315480] LustreError: 164775:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4793.326832] LustreError: 164775:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 4796.316826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.749883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4801.760403] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4809.924950] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4815.481094] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4821.581767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4821.940715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4828.113172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4828.527800] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4836.422534] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4845.177315] Lustre: Failing over lustre-MDT0000 [ 4845.496778] Lustre: server umount lustre-MDT0000 complete [ 4846.563438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4846.567497] LustreError: 164770:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4846.571633] Lustre: Skipped 2 previous similar messages [ 4846.584724] LustreError: 164770:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 4851.689543] LustreError: 164772:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.717693] LustreError: 164772:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 4856.804435] LustreError: 166582:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4856.829314] LustreError: 166582:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 4857.611545] LustreError: 167355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.55@tcp arrived at 1787850761 with bad export cookie 16776964470041696998 [ 4857.637928] LustreError: 167355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4861.930101] Lustre: 164087:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850750/real 1787850750] req@00000000a545a1d2 x1874697309415808/t0(0) o400->MGC192.168.206.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787850766 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4861.936016] LustreError: 164770:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4861.958825] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4862.007746] LustreError: 164770:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 4864.395363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.168603] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe8d3c04c1e2f5b4f [ 4872.180646] LustreError: 165505:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4872.182092] Lustre: MGC192.168.206.155@tcp: Connection restored to 0@lo (at 0@lo) [ 4872.208300] LustreError: 165505:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 5 previous similar messages [ 4872.516310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4872.570293] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4877.051890] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4877.807682] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4877.809771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4879.518829] Lustre: lustre-MDT0000: Denying connection for new client efd6f030-2038-4bde-a6d4-1b101dece707 (at 192.168.206.55@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 4888.038812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4888.047197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4888.053709] Lustre: Skipped 1 previous similar message [ 4893.155679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4893.168540] Lustre: Skipped 1 previous similar message [ 4898.284664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4898.298722] Lustre: Skipped 1 previous similar message [ 4902.367118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4902.555172] Lustre: server umount lustre-OST0000 complete [ 4905.907792] LustreError: 168508:0:(ldlm_lib.c:3051:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4905.914427] Lustre: 167856:0:(ldlm_lib.c:2451:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4905.924541] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4905.939516] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4905.951868] LustreError: 167856:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007462fb32 x1874697309436416/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4905.963397] LustreError: 167856:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4905.969231] LustreError: 167856:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4905.977572] Lustre: lustre-MDT0000: Recovery over after 0:28, of 2 clients 0 recovered and 2 were evicted. [ 4906.255164] Lustre: server umount lustre-MDT0000 complete [ 4908.512257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4908.524164] LustreError: 164772:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4909.460178] LustreError: 167355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850813 with bad export cookie 16776964470041697103 [ 4909.466918] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4909.483075] LustreError: 167355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4915.926198] Lustre: server umount lustre-MDT0001 complete [ 4926.222322] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 4930.674755] Key type lgssc unregistered [ 4931.072049] LNet: 169266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4931.081736] LNetError: 169266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4931.098959] LNet: Removed LNI 192.168.206.155@tcp [ 4932.487331] Key type .llcrypt unregistered [ 4932.491119] Key type ._llcrypt unregistered [ 4964.936950] Key type ._llcrypt registered [ 4964.940714] Key type .llcrypt registered [ 4965.073480] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4965.932947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4965.968378] alg: No test for adler32 (adler32-zlib) [ 4967.064523] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 4967.333823] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 4969.063150] Key type lgssc registered [ 4970.249951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4979.268979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4979.293082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4980.865899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4985.193795] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4990.468818] LustreError: 170678:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4990.502107] LustreError: 170678:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 4993.482663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4993.772992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4997.990088] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5004.771510] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5010.014489] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5016.235084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5016.549865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5022.534880] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5025.835373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5030.075710] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5036.005866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.022448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5036.036823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5036.058538] Lustre: Skipped 1 previous similar message [ 5041.122956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5041.129225] Lustre: Skipped 1 previous similar message [ 5041.956541] Lustre: server umount lustre-OST0000 complete [ 5046.241679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.255468] Lustre: Skipped 1 previous similar message [ 5046.263675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5051.366970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5051.371377] Lustre: Skipped 2 previous similar messages [ 5051.977756] Lustre: server umount lustre-MDT0000 complete [ 5055.397089] LustreError: 170659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850959 with bad export cookie 9162654457142260621 [ 5055.412790] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5055.415827] LustreError: 170659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5055.759294] Lustre: server umount lustre-MDT0001 complete [ 5066.056969] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5068.922459] Key type lgssc unregistered [ 5069.161134] LNet: 174212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5069.165845] LNetError: 174212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5069.185150] LNet: Removed LNI 192.168.206.155@tcp [ 5070.032530] Key type .llcrypt unregistered [ 5070.036898] Key type ._llcrypt unregistered [ 5096.450704] Key type ._llcrypt registered [ 5096.453411] Key type .llcrypt registered [ 5096.557777] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5097.512161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5097.533995] alg: No test for adler32 (adler32-zlib) [ 5098.525187] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 5098.731467] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 5100.383150] Key type lgssc registered [ 5101.394927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5107.310281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5107.332070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5108.642056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5112.013425] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5117.922513] LustreError: 175893:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5117.947029] LustreError: 175893:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 5119.195605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5119.518264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5123.402527] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5130.389711] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5135.738728] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5142.020564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5142.345206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5148.668203] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5153.718630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5155.917571] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5164.007667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.007784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5164.024194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5164.041134] Lustre: Skipped 1 previous similar message [ 5167.013883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5167.025372] Lustre: Skipped 1 previous similar message [ 5168.756224] Lustre: server umount lustre-OST0000 complete [ 5172.194446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5172.197577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5172.206325] Lustre: Skipped 2 previous similar messages [ 5177.322351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5177.324449] Lustre: Skipped 2 previous similar messages [ 5177.865985] Lustre: server umount lustre-MDT0000 complete [ 5181.237694] LustreError: 178675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851085 with bad export cookie 10129344184191119459 [ 5181.248086] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5181.253603] LustreError: 178675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5181.570483] Lustre: server umount lustre-MDT0001 complete [ 5191.486224] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5194.220918] Key type lgssc unregistered [ 5194.485912] LNet: 179431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5194.494362] LNetError: 179431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5194.516235] LNet: Removed LNI 192.168.206.155@tcp [ 5195.390992] Key type .llcrypt unregistered [ 5195.394886] Key type ._llcrypt unregistered [ 5214.895333] Key type ._llcrypt registered [ 5214.898710] Key type .llcrypt registered [ 5214.989863] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5216.034529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5216.099828] alg: No test for adler32 (adler32-zlib) [ 5217.170286] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 5217.388063] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 5219.087223] Key type lgssc registered [ 5220.251555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5229.752905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5229.775604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5231.086842] Lustre: *** cfs_fail_loc=135, val=0*** [ 5231.164972] LustreError: 180934:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5231.174508] LustreError: 180934:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.155@tcp: cfg command failed: rc = -2 [ 5231.191876] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5231.191876] [ 5231.210364] LustreError: MGC192.168.206.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5231.222792] LustreError: 180895:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5231.236091] LustreError: 180895:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5231.244584] LustreError: 180895:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5231.249097] LustreError: 180895:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5231.316701] Lustre: server umount lustre-MDT0000 complete [ 5231.321873] LustreError: 180895:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5237.634646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5237.904209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5241.862618] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5247.463779] LustreError: 181306:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5247.475258] LustreError: 181306:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 5249.067993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5249.342787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5252.915774] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5258.310990] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5263.094479] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5264.866700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5264.876762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5264.888517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5267.937437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5267.938686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5267.957164] Lustre: Skipped 1 previous similar message [ 5273.060130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5273.068295] Lustre: Skipped 1 previous similar message [ 5279.202682] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5279.207437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.222138] Lustre: Skipped 3 previous similar messages [ 5279.352908] Lustre: server umount lustre-MDT0000 complete [ 5282.218057] LustreError: 181285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851186 with bad export cookie 13173319539232292454 [ 5282.224452] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5282.236653] LustreError: 181285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5282.512096] Lustre: server umount lustre-MDT0001 complete [ 5290.989432] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5293.067843] Key type lgssc unregistered [ 5293.289829] LNet: 183629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5293.302689] LNetError: 183629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5293.322847] LNet: Removed LNI 192.168.206.155@tcp [ 5293.980664] Key type .llcrypt unregistered [ 5293.983910] Key type ._llcrypt unregistered [ 5311.178941] Key type ._llcrypt registered [ 5311.182500] Key type .llcrypt registered [ 5311.339228] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5311.684613] Key type .llcrypt unregistered [ 5311.689981] Key type ._llcrypt unregistered [ 5324.922579] Key type ._llcrypt registered [ 5324.926802] Key type .llcrypt registered [ 5325.040377] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5325.956215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5325.977166] alg: No test for adler32 (adler32-zlib) [ 5327.011270] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 5327.256728] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 5328.911153] Key type lgssc registered [ 5330.308669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5338.240262] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5338.259912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5353.887487] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5359.009243] LustreError: 185743:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [ 5385.760855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5392.739348] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5400.440196] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5407.458737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.878112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5407.938786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5412.137774] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5413.355317] LustreError: 185768:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.377650] LustreError: 185768:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 5418.467859] LustreError: 185769:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5418.881691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5419.280323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5423.683968] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5430.308852] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5435.336940] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5437.937270] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5450.208614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.214552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.237580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5451.239915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5451.244352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.797101] Lustre: server umount lustre-OST0000 complete [ 5460.448454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5460.455605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.478953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5460.492229] Lustre: Skipped 1 previous similar message [ 5465.549499] Lustre: server umount lustre-MDT0000 complete [ 5466.599795] LustreError: 186781:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5466.624250] LustreError: 186781:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 5468.980315] LustreError: 185750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851373 with bad export cookie 8739379135243689203 [ 5468.986805] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5468.993500] LustreError: 185750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5469.264737] Lustre: server umount lustre-MDT0001 complete [ 5478.896900] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5481.468559] Key type lgssc unregistered [ 5481.775670] LNet: 189759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5481.791829] LNetError: 189759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5481.818872] LNet: Removed LNI 192.168.206.155@tcp [ 5482.756353] Key type .llcrypt unregistered [ 5482.760836] Key type ._llcrypt unregistered [ 5504.851298] Key type ._llcrypt registered [ 5504.855389] Key type .llcrypt registered [ 5504.943771] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5506.020330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5506.123712] alg: No test for adler32 (adler32-zlib) [ 5507.299049] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 5507.559562] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 5509.247187] Key type lgssc registered [ 5510.293109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5517.425977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5517.441764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.854065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5523.010336] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5528.039412] LustreError: 191172:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5528.060364] LustreError: 191172:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 5530.489285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.824942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5534.955229] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5542.065941] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5548.046277] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5554.444996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5554.784495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5560.275651] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5563.005252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5567.212610] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5572.300984] Lustre: Failing over lustre-MDT0000 [ 5572.867693] Lustre: server umount lustre-MDT0000 complete [ 5573.604163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.630055] LustreError: 191169:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.634440] Lustre: Skipped 2 previous similar messages [ 5573.666061] LustreError: 191169:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 5578.722458] LustreError: 192236:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5578.733895] LustreError: 192236:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 5583.842914] LustreError: 191167:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.880726] LustreError: 191167:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 5588.959101] Lustre: 190484:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851477/real 1787851477] req@0000000003a03a40 x1874698081166592/t0(0) o400->MGC192.168.206.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787851493 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5588.967060] LustreError: 192237:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5588.978635] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5589.016359] LustreError: 192237:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 5590.634776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5599.203441] LustreError: 190482:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003e3531cc x1874698081171456/t0(0) o250->MGC192.168.206.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5599.228982] LustreError: 192236:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5599.266012] LustreError: 192236:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 9 previous similar messages [ 5599.448324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.518456] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5599.550544] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5604.359253] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5604.861939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5604.895685] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5604.968578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5609.524910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5619.134779] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5630.499202] Lustre: server umount lustre-OST0000 complete [ 5635.553953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.557283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5635.569755] Lustre: Skipped 1 previous similar message [ 5635.588319] Lustre: Skipped 1 previous similar message [ 5640.672482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5640.689859] Lustre: Skipped 1 previous similar message [ 5644.885377] Lustre: server umount lustre-MDT0000 complete [ 5645.794518] LustreError: 192236:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5645.823499] LustreError: 192236:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 5646.824302] Lustre: 190483:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851534/real 1787851534] req@000000004e02b689 x1874698081199488/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787851550 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5646.859239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.927476] Lustre: 190485:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851537/real 1787851537] req@000000002ed4110d x1874698081200384/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787851553 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5648.979843] LustreError: 191152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851553 with bad export cookie 4757940441570503366 [ 5648.987058] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5648.992107] LustreError: 191152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5649.325710] Lustre: server umount lustre-MDT0001 complete [ 5660.656838] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5663.585278] Key type lgssc unregistered [ 5663.775457] LNet: 196111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5663.791992] LNetError: 196111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5663.820653] LNet: Removed LNI 192.168.206.155@tcp [ 5664.454495] Key type .llcrypt unregistered [ 5664.460478] Key type ._llcrypt unregistered [ 5686.397519] Key type ._llcrypt registered [ 5686.399286] Key type .llcrypt registered [ 5686.490841] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5687.174586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5687.301437] alg: No test for adler32 (adler32-zlib) [ 5688.285401] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 5688.504098] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 5690.255182] Key type lgssc registered [ 5691.709624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5700.508757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5700.519878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.973565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5705.496182] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5712.355565] LustreError: 197519:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5712.371412] LustreError: 197519:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 5712.981584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5713.257956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5716.283403] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5721.936346] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5726.471837] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5732.143379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5732.513050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.890638] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5744.924925] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5747.887767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5752.920914] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5761.664313] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5785.104684] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5794.273888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5794.280342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5794.292666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5794.295872] Lustre: Skipped 1 previous similar message [ 5799.347762] Lustre: server umount lustre-OST0000 complete [ 5804.512517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.513780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5804.527530] Lustre: Skipped 1 previous similar message [ 5804.538494] Lustre: Skipped 1 previous similar message [ 5809.120643] Lustre: server umount lustre-MDT0000 complete [ 5809.632708] LustreError: 198862:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5809.650622] LustreError: 198862:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 5813.949217] LustreError: 197500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851718 with bad export cookie 7387873856257053844 [ 5813.951276] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5813.955175] LustreError: 197500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5814.172474] Lustre: server umount lustre-MDT0001 complete [ 5829.431482] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5832.559620] Key type lgssc unregistered [ 5832.889314] LNet: 201396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5832.906167] LNetError: 201396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5832.935781] LNet: Removed LNI 192.168.206.155@tcp [ 5833.985081] Key type .llcrypt unregistered [ 5833.989713] Key type ._llcrypt unregistered [ 5850.960643] Key type ._llcrypt registered [ 5850.962167] Key type .llcrypt registered [ 5851.082328] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5851.856423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5851.924013] alg: No test for adler32 (adler32-zlib) [ 5852.940866] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 5853.205092] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 5854.903958] Key type lgssc registered [ 5855.936878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5864.052732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5864.083217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.590217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5869.937534] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5875.195814] LustreError: 202663:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5875.222216] LustreError: 202663:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 5878.630647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.032630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5884.201112] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5892.061361] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5897.569585] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5904.215339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5904.645843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5910.845211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5911.148373] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5919.321591] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5925.930396] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5936.975956] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5946.967309] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5952.892916] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5967.329208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.337914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5967.377960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5967.402804] Lustre: Skipped 1 previous similar message [ 5970.262405] Lustre: server umount lustre-OST0000 complete [ 5976.545899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5976.561763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.573133] Lustre: Skipped 1 previous similar message [ 5976.585584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5977.574070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.592155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5977.603526] Lustre: Skipped 1 previous similar message [ 5980.978372] Lustre: server umount lustre-MDT0000 complete [ 5982.688939] LustreError: 202662:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.722844] LustreError: 202662:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 5984.537105] LustreError: 202642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851888 with bad export cookie 14803943389689279307 [ 5984.537567] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5984.547986] LustreError: 202642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5985.102241] Lustre: server umount lustre-MDT0001 complete [ 5997.240153] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5999.793563] Key type lgssc unregistered [ 6000.054627] LNet: 207639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6000.058689] LNetError: 207639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6000.071634] LNet: Removed LNI 192.168.206.155@tcp [ 6000.969110] Key type .llcrypt unregistered [ 6000.971094] Key type ._llcrypt unregistered [ 6023.685593] Key type ._llcrypt registered [ 6023.688484] Key type .llcrypt registered [ 6023.785525] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6024.789294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6024.810083] alg: No test for adler32 (adler32-zlib) [ 6025.874069] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 6026.197778] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 6027.927175] Key type lgssc registered [ 6029.035572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6035.789209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6035.803755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6037.082241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.766532] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6047.207277] LustreError: 209053:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.239425] LustreError: 209053:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 6050.149304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.510961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6055.387881] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6064.066918] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6070.586511] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6078.941377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6079.315605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6082.417162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6086.235661] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6094.185176] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6108.128252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6108.143747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6108.165353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6111.218179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6111.219527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6111.241042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6111.261283] Lustre: Skipped 1 previous similar message [ 6113.386158] Lustre: server umount lustre-OST0000 complete [ 6121.442352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.448863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6121.477415] Lustre: Skipped 1 previous similar message [ 6121.511046] Lustre: Skipped 1 previous similar message [ 6123.841692] Lustre: server umount lustre-MDT0000 complete [ 6126.570964] LustreError: 210058:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6126.603705] LustreError: 210058:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 6127.532653] LustreError: 209034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852031 with bad export cookie 2085060829909613695 [ 6127.538659] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6127.542031] LustreError: 209034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6127.949843] Lustre: server umount lustre-MDT0001 complete [ 6139.836813] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 6142.647550] Key type lgssc unregistered [ 6142.965986] LNet: 212742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6142.969530] LNetError: 212742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6143.005267] LNet: Removed LNI 192.168.206.155@tcp [ 6143.921525] Key type .llcrypt unregistered [ 6143.923336] Key type ._llcrypt unregistered [ 6162.764685] Key type ._llcrypt registered [ 6162.767202] Key type .llcrypt registered [ 6162.885460] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6164.142421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6164.213842] alg: No test for adler32 (adler32-zlib) [ 6165.320669] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 6165.556069] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 6167.311152] Key type lgssc registered [ 6168.455383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6175.425485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6175.442484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.885065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.024210] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6186.466388] LustreError: 214006:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6186.490623] LustreError: 214006:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 6188.975030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6189.275649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6193.282523] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6202.586267] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6210.106217] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6217.597976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6218.063386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6221.198791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6224.320662] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6231.565343] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6249.955430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6249.959280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6249.975656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6254.772365] Lustre: server umount lustre-OST0000 complete [ 6260.193770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6260.219731] Lustre: Skipped 1 previous similar message [ 6260.227208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6260.250898] Lustre: Skipped 2 previous similar messages [ 6265.314125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6265.326252] Lustre: Skipped 1 previous similar message [ 6266.302171] Lustre: server umount lustre-MDT0000 complete [ 6270.433547] LustreError: 214007:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.490201] LustreError: 214007:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 6271.093861] LustreError: 213986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852175 with bad export cookie 211552527647513288 [ 6271.101995] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.107826] LustreError: 213986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6271.566117] Lustre: server umount lustre-MDT0001 complete [ 6285.042164] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 6287.648587] Key type lgssc unregistered [ 6287.935831] LNet: 217692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6287.945793] LNetError: 217692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6287.963633] LNet: Removed LNI 192.168.206.155@tcp [ 6288.798912] Key type .llcrypt unregistered [ 6288.803968] Key type ._llcrypt unregistered [ 6307.124870] Key type ._llcrypt registered [ 6307.128912] Key type .llcrypt registered [ 6307.270309] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6308.582163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6308.618948] alg: No test for adler32 (adler32-zlib) [ 6309.713423] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 6310.046209] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 6311.791624] Key type lgssc registered [ 6312.830489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6321.118945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6321.136883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.638927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6327.610913] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6332.896156] LustreError: 218958:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.921404] LustreError: 218958:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 6335.499648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.868104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6340.319198] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6349.008672] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6354.972141] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6362.303091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6362.773225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6367.935698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6370.109475] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6378.136241] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6386.637257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6386.756322] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6386.911483] LustreError: 220778:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6387.090199] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6393.460163] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6400.654914] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6401.965114] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6414.090865] Lustre: Permanently deactivating lustre-OST0001 [ 6414.130749] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6421.595242] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6426.173613] Lustre: Permanently reactivating lustre-OST0001 [ 6426.200985] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6426.212241] Lustre: Skipped 2 previous similar messages [ 6431.319436] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6431.340136] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6431.350883] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6431.360676] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6431.406074] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6436.359518] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.372231] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6436.374144] Lustre: Skipped 2 previous similar messages [ 6436.390104] Lustre: Skipped 1 previous similar message [ 6437.726944] Lustre: server umount lustre-OST0001 complete [ 6441.463088] LustreError: 221797:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6441.472374] LustreError: 221797:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 6441.952681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6441.962842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.987771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6445.026222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6445.043599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.069546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6446.576520] LustreError: 220780:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6446.601658] LustreError: 220780:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 6451.684461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6451.685525] LustreError: 221797:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6451.693468] Lustre: Skipped 3 previous similar messages [ 6451.711099] LustreError: 221797:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 6456.287140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6456.887867] Lustre: server umount lustre-OST0000 complete [ 6461.929141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6461.936700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6461.944204] Lustre: Skipped 1 previous similar message [ 6461.952973] Lustre: Skipped 1 previous similar message [ 6468.081466] Lustre: server umount lustre-MDT0000 complete [ 6472.006758] LustreError: 218939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852376 with bad export cookie 15648590650915140821 [ 6472.014398] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6472.017607] LustreError: 218939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6472.403929] Lustre: server umount lustre-MDT0001 complete [ 6485.996681] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 6490.006083] Key type lgssc unregistered [ 6490.393752] LNet: 224013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6490.408772] LNetError: 224013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6490.435847] LNet: Removed LNI 192.168.206.155@tcp [ 6491.458471] Key type .llcrypt unregistered [ 6491.461055] Key type ._llcrypt unregistered [ 6533.459690] Key type ._llcrypt registered [ 6533.469151] Key type .llcrypt registered [ 6533.651779] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6535.628389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6535.666893] alg: No test for adler32 (adler32-zlib) [ 6536.832201] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 6537.176664] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 6538.863470] Key type lgssc registered [ 6540.552949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6548.534634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6548.576082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6550.088216] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6550.114461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6550.231589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6555.920833] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6563.904694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6563.966397] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6563.993753] Lustre: 226221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6564.180672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6570.045920] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6578.341384] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6585.127559] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6591.624173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6591.803522] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6591.978717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6598.277190] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6599.860424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6609.083405] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6612.088606] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6617.448915] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6626.834402] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6632.463911] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6641.349771] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6658.438761] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6658.449907] Lustre: Skipped 1 previous similar message [ 6694.868193] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6694.872288] Lustre: Skipped 4 previous similar messages [ 6766.936460] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6766.946276] Lustre: Skipped 9 previous similar messages [ 6774.243760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6774.245573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6774.264229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6774.270594] Lustre: Skipped 1 previous similar message [ 6779.371352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6779.393073] Lustre: Skipped 1 previous similar message [ 6779.726278] Lustre: server umount lustre-OST0000 complete [ 6784.484060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6784.486670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6784.507879] LustreError: Skipped 1 previous similar message [ 6784.530095] Lustre: Skipped 1 previous similar message [ 6784.530758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6789.604917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6789.620197] Lustre: Skipped 2 previous similar messages [ 6789.851163] Lustre: server umount lustre-MDT0000 complete [ 6794.945791] LustreError: 229172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852699 with bad export cookie 6627625426041857748 [ 6794.950597] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6794.969472] LustreError: 229172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6795.018755] LustreError: 225545:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 6796.120452] Lustre: server umount lustre-MDT0001 complete [ 6810.260571] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 6813.669171] Key type lgssc unregistered [ 6814.074450] LNet: 231001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6814.092963] LNetError: 231001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6814.122809] LNet: Removed LNI 192.168.206.155@tcp [ 6815.608419] Key type .llcrypt unregistered [ 6815.616655] Key type ._llcrypt unregistered [ 6839.597065] Key type ._llcrypt registered [ 6839.603454] Key type .llcrypt registered [ 6839.716228] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6840.665303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6840.813229] alg: No test for adler32 (adler32-zlib) [ 6841.844761] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 6842.046358] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 6843.736925] Key type lgssc registered [ 6844.798239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6852.098159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6852.124748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6853.532440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6858.187336] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6863.847245] LustreError: 232418:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6863.885783] LustreError: 232418:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 6865.303174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6865.765335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6869.616234] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6875.650982] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6882.200952] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6889.614294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6890.061187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6895.877637] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6898.228772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6902.457217] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6921.699194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6921.708589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.745636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6923.759181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6923.767568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6923.790297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6927.957643] Lustre: server umount lustre-OST0000 complete [ 6932.448528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6932.460988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6932.482383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6932.492984] Lustre: Skipped 1 previous similar message [ 6938.658349] Lustre: server umount lustre-MDT0000 complete [ 6939.114506] LustreError: 232412:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6939.146034] LustreError: 232412:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 6942.460608] LustreError: 232398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852846 with bad export cookie 9394689316813317591 [ 6942.465554] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6942.483150] LustreError: 232398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6942.868140] Lustre: server umount lustre-MDT0001 complete [ 6952.915590] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 6955.700992] Key type lgssc unregistered [ 6956.110523] LNet: 236109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6956.121320] LNetError: 236109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6956.142605] LNet: Removed LNI 192.168.206.155@tcp [ 6956.969202] Key type .llcrypt unregistered [ 6956.974066] Key type ._llcrypt unregistered [ 6977.905902] Key type ._llcrypt registered [ 6977.907545] Key type .llcrypt registered [ 6978.025698] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6978.572054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6978.593633] alg: No test for adler32 (adler32-zlib) [ 6979.639992] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 6979.902966] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 6981.591289] Key type lgssc registered [ 6982.711826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7253.486325] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7256.521857] Key type lgssc unregistered [ 7256.803600] LNet: 238134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7256.810993] LNetError: 238134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7256.832667] LNet: Removed LNI 192.168.206.155@tcp [ 7257.706437] Key type .llcrypt unregistered [ 7257.708331] Key type ._llcrypt unregistered [ 7281.761444] Key type ._llcrypt registered [ 7281.763154] Key type .llcrypt registered [ 7281.851756] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 7296.175122] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7297.122583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7297.149132] alg: No test for adler32 (adler32-zlib) [ 7298.294861] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 7298.690266] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7300.359478] Key type lgssc registered [ 7301.388491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7329.372865] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7093 sec ========== 13:53:52 (1787853232) [ 7331.080613] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:53:54 (1787853234) === [ 7334.496615] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:53:57 (1787853237) === [ 7370.643274] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7373.422490] Key type lgssc unregistered [ 7373.676854] LNet: 243378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7373.685786] LNetError: 243378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7373.709685] LNet: Removed LNI 192.168.206.155@tcp [ 7374.697754] Key type .llcrypt unregistered [ 7374.700968] Key type ._llcrypt unregistered