[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506655268 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002188] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.003911] kvm-guest: setup PV IPIs [ 0.005736] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007012] pid_max: default: 32768 minimum: 301 [ 0.009135] LSM: Security Framework initializing [ 0.010064] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.012080] *** VALIDATE selinux *** [ 0.019350] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024113] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026080] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028109] *** VALIDATE tmpfs *** [ 0.029438] *** VALIDATE proc *** [ 0.030227] *** VALIDATE cgroup *** [ 0.031000] *** VALIDATE cgroup2 *** [ 0.032103] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033125] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035035] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.038766] debug: unmapping init [mem 0xffffffffaac59000-0xffffffffaac60fff] [ 0.040178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041559] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042020] ... version: 2 [ 0.042838] ... bit width: 48 [ 0.043011] ... generic registers: 4 [ 0.043784] ... value mask: 0000ffffffffffff [ 0.044012] ... max period: 00007fffffffffff [ 0.045012] ... fixed-purpose events: 3 [ 0.045781] ... event mask: 000000070000000f [ 0.046263] rcu: Hierarchical SRCU implementation. [ 0.048160] smp: Bringing up secondary CPUs ... [ 0.049407] x86: Booting SMP configuration: [ 0.050032] .... node #0, CPUs: #1 #2 #3 [ 0.058186] smp: Brought up 1 node, 4 CPUs [ 0.060018] smpboot: Max logical packages: 1 [ 0.061018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.098736] node 0 deferred pages initialised in 36ms [ 0.102012] devtmpfs: initialized [ 0.102922] x86/mm: Memory block size: 128MB [ 0.104633] gcov: version magic: 0x41383552 [ 0.105577] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.106062] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.107213] pinctrl core: initialized pinctrl subsystem [ 0.108140] [ 0.108422] ************************************************************* [ 0.109011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.110008] ** ** [ 0.111008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.112012] ** ** [ 0.113011] ** This means that this kernel is built to expose internal ** [ 0.114011] ** IOMMU data structures, which may compromise security on ** [ 0.115011] ** your system. ** [ 0.116009] ** ** [ 0.117009] ** If you see this message and you are not debugging the ** [ 0.118012] ** kernel, report this immediately to your vendor! ** [ 0.119010] ** ** [ 0.120012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.121010] ************************************************************* [ 0.122518] NET: Registered protocol family 16 [ 0.123346] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.124041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.125046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.126349] cpuidle: using governor menu [ 0.127605] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.129421] PCI: Using configuration type 1 for base access [ 0.130117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.138054] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.139015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.140133] cryptd: max_cpu_qlen set to 1000 [ 0.144242] ACPI: Added _OSI(Module Device) [ 0.145009] ACPI: Added _OSI(Processor Device) [ 0.146010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.146997] ACPI: Added _OSI(Processor Aggregator Device) [ 0.150012] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.154528] ACPI: Interpreter enabled [ 0.155045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.155949] ACPI: Using IOAPIC for interrupt routing [ 0.156076] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.157258] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.166768] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.167037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.168016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.169086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.170667] acpiphp: Slot [2] registered [ 0.172167] acpiphp: Slot [5] registered [ 0.174136] acpiphp: Slot [6] registered [ 0.176128] acpiphp: Slot [7] registered [ 0.177129] acpiphp: Slot [8] registered [ 0.179125] acpiphp: Slot [9] registered [ 0.180118] acpiphp: Slot [10] registered [ 0.182117] acpiphp: Slot [3] registered [ 0.183123] acpiphp: Slot [4] registered [ 0.185107] acpiphp: Slot [11] registered [ 0.186116] acpiphp: Slot [12] registered [ 0.188095] acpiphp: Slot [13] registered [ 0.189089] acpiphp: Slot [14] registered [ 0.191049] acpiphp: Slot [15] registered [ 0.192040] acpiphp: Slot [16] registered [ 0.193093] acpiphp: Slot [17] registered [ 0.193994] acpiphp: Slot [18] registered [ 0.195104] acpiphp: Slot [19] registered [ 0.197113] acpiphp: Slot [20] registered [ 0.198096] acpiphp: Slot [21] registered [ 0.199132] acpiphp: Slot [22] registered [ 0.199989] acpiphp: Slot [23] registered [ 0.200102] acpiphp: Slot [24] registered [ 0.201104] acpiphp: Slot [25] registered [ 0.202058] acpiphp: Slot [26] registered [ 0.203044] acpiphp: Slot [27] registered [ 0.204025] acpiphp: Slot [28] registered [ 0.205022] acpiphp: Slot [29] registered [ 0.205976] acpiphp: Slot [30] registered [ 0.207077] acpiphp: Slot [31] registered [ 0.208009] PCI host bridge to bus 0000:00 [ 0.208904] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.210019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.212015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.213019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.215018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.217023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.218143] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.220551] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.222902] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.229967] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.233058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.237017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.238014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.241021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.243737] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.247315] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.251071] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.254403] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.260018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.273013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.277000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.282555] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.287021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.299017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.321016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.331203] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.336020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.342018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.356023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.363691] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.369017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.373014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.382016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.393029] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.401024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.409020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.425019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.433336] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.439018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.445018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.456017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.465583] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.470015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.475017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.486020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.495000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.495000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.497428] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.499238] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.500359] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.505209] iommu: Default domain type: Passthrough [ 0.506494] SCSI subsystem initialized [ 0.507159] ACPI: bus type USB registered [ 0.508117] usbcore: registered new interface driver usbfs [ 0.510066] usbcore: registered new interface driver hub [ 0.511059] usbcore: registered new device driver usb [ 0.513175] pps_core: LinuxPPS API ver. 1 registered [ 0.514009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.518086] PTP clock support registered [ 0.520042] EDAC MC: Ver: 3.0.0 [ 0.521106] PCI: Using ACPI for IRQ routing [ 0.522405] NetLabel: Initializing [ 0.523010] NetLabel: domain hash size = 128 [ 0.525012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.527083] NetLabel: unlabeled traffic allowed by default [ 0.529135] vgaarb: loaded [ 0.530270] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.532017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.539044] clocksource: Switched to clocksource kvm-clock [ 0.653297] VFS: Disk quotas dquot_6.6.0 [ 0.654906] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.657205] *** VALIDATE ramfs *** [ 0.658619] *** VALIDATE hugetlbfs *** [ 0.660397] pnp: PnP ACPI init [ 0.663072] pnp: PnP ACPI: found 6 devices [ 0.679288] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.682748] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.684519] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.686795] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.689321] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.691904] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.694681] NET: Registered protocol family 2 [ 0.696840] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.701804] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.705435] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.710222] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.713306] TCP: Hash tables configured (established 65536 bind 65536) [ 0.715663] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.718171] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.720523] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.723297] NET: Registered protocol family 1 [ 0.726295] RPC: Registered named UNIX socket transport module. [ 0.728121] RPC: Registered udp transport module. [ 0.729488] RPC: Registered tcp transport module. [ 0.730628] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.732625] NET: Registered protocol family 44 [ 0.733723] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.735072] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.736465] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.737807] PCI: CLS 0 bytes, default 64 [ 0.738868] Unpacking initramfs... [ 1.979980] debug: unmapping init [mem 0xffff8b73bcc54000-0xffff8b73bffbffff] [ 1.982869] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.984577] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.986669] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.472316] Initialise system trusted keyrings [ 2.473678] Key type blacklist registered [ 2.474905] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.483682] zbud: loaded [ 2.486836] *** VALIDATE nfs *** [ 2.488058] *** VALIDATE nfs4 *** [ 2.489561] pstore: using deflate compression [ 2.492519] Platform Keyring initialized [ 2.594287] NET: Registered protocol family 38 [ 2.595663] Key type asymmetric registered [ 2.596829] Asymmetric key parser 'x509' registered [ 2.598069] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.600442] io scheduler mq-deadline registered [ 2.602076] io scheduler kyber registered [ 2.603661] io scheduler bfq registered [ 2.605059] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.607853] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.612073] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.615269] ACPI: Power Button [PWRF] [ 2.619670] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.626759] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.637808] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.643989] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.654235] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.679832] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.705509] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.708911] Non-volatile memory driver v1.3 [ 2.710087] Linux agpgart interface v0.103 [ 2.738660] virtio_blk virtio1: [vda] 134872 512-byte logical blocks (69.1 MB/65.9 MiB) [ 2.740841] vda: detected capacity change from 0 to 69054464 [ 2.753272] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.755509] vdb: detected capacity change from 0 to 1073741824 [ 2.767637] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.769759] vdc: detected capacity change from 0 to 2621440000 [ 2.782408] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.784678] vdd: detected capacity change from 0 to 2621440000 [ 2.796081] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.798215] vde: detected capacity change from 0 to 4294967296 [ 2.809649] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.811725] vdf: detected capacity change from 0 to 4294967296 [ 2.816537] libphy: Fixed MDIO Bus: probed [ 2.821859] usbcore: registered new interface driver usbserial_generic [ 2.824357] usbserial: USB Serial support registered for generic [ 2.827193] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.831400] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.833216] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.835590] mousedev: PS/2 mouse device common for all mice [ 2.837739] rtc_cmos 00:05: RTC can wake from S4 [ 2.840176] rtc_cmos 00:05: registered as rtc0 [ 2.842304] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.845516] intel_pstate: CPU model not supported [ 2.848211] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.851197] hid: raw HID events driver (C) Jiri Kosina [ 2.854092] usbcore: registered new interface driver usbhid [ 2.855374] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.856715] usbhid: USB HID core driver [ 2.861041] drop_monitor: Initializing network drop monitor service [ 2.861184] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.863409] Initializing XFRM netlink socket [ 2.867510] NET: Registered protocol family 10 [ 2.870654] Segment Routing with IPv6 [ 2.871875] NET: Registered protocol family 17 [ 2.873945] mpls_gso: MPLS GSO support [ 2.879598] RAS: Correctable Errors collector initialized. [ 2.881247] AVX version of gcm_enc/dec engaged. [ 2.883031] AES CTR mode by8 optimization enabled [ 2.949032] sched_clock: Marking stable (2949007717, 0)->(3816656092, -867648375) [ 2.952337] registered taskstats version 1 [ 2.954407] Loading compiled-in X.509 certificates [ 2.956561] zswap: loaded using pool lzo/zbud [ 2.979931] Key type big_key registered [ 2.993983] Key type encrypted registered [ 2.995682] ima: No TPM chip found, activating TPM-bypass! [ 2.997710] ima: Allocated hash algorithm: sha1 [ 2.999192] ima: No architecture policies found [ 3.001027] evm: Initialising EVM extended attributes: [ 3.002964] evm: security.selinux [ 3.004181] evm: security.ima [ 3.005486] evm: security.capability [ 3.006780] evm: HMAC attrs: 0x1 [ 3.009198] rtc_cmos 00:05: setting system clock to 2026-06-01 17:15:29 UTC (1780334129) [ 3.014543] debug: unmapping init [mem 0xffffffffabc03000-0xffffffffabdfffff] [ 3.017523] debug: unmapping init [mem 0xffffffffaa982000-0xffffffffaac58fff] [ 3.027103] Write protecting the kernel read-only data: 28672k [ 3.030417] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 3.033329] debug: unmapping init [mem 0xffffffffa9914000-0xffffffffa99fffff] [ 3.064926] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.073283] systemd[1]: Detected virtualization kvm. [ 3.074637] systemd[1]: Detected architecture x86-64. [ 3.075779] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.098619] systemd[1]: No hostname configured. [ 3.100042] systemd[1]: Set hostname to . [ 3.102042] random: systemd: uninitialized urandom read (16 bytes read) [ 3.104588] systemd[1]: Initializing machine ID from random generator. [ 3.144020] random: ln: uninitialized urandom read (6 bytes read) [ 3.214463] random: systemd: uninitialized urandom read (16 bytes read) [ 3.216872] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.219995] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.224776] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.690238] device-mapper: uevent: version 1.0.3 [ 3.692234] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 4.414743] random: fast init done [ 4.416500] virtio_net virtio0 ens2: renamed from eth0 [ 4.435921] scsi host0: ata_piix [ 4.465239] scsi host1: ata_piix [ 4.470895] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.474078] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.267264] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.517309] random: crng init done [ 9.519126] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.925550] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.160891] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.442161] SELinux: Disabled at runtime. [ 11.506882] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.515502] systemd[1]: Detected virtualization kvm. [ 11.517648] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.995243] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.998952] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.004866] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.008135] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.012446] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.020571] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.025329] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Jour[ 12.227856] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS nal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.501633] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.728686] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.830905] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.944605] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.959046] EDAC sbridge: Ver: 1.1.2 [ 14.678355] Key type dns_resolver registered [ 14.989399] NFS: Registering the id_resolver key type [ 14.991576] Key type id_resolver registered [ 14.993484] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg129-server login: [ 56.273704] libcfs: loading out-of-tree module taints kernel. [ 56.333151] Key type ._llcrypt registered [ 56.337776] Key type .llcrypt registered [ 56.493230] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 78.906600] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 81.044887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 81.064422] alg: No test for adler32 (adler32-zlib) [ 82.557052] hrtimer: interrupt took 2377013 ns [ 82.810902] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 84.259518] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 86.159229] Key type lgssc registered [ 88.315098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 106.748062] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 157.819810] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 172.616611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 172.647142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.947477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 173.983789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 174.090704] Lustre: lustre-MDT0000: new disk, initializing [ 174.166070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.185822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.441291] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.442127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.531609] Lustre: 6506:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 192.557577] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 192.569917] Lustre: Skipped 1 previous similar message [ 192.746847] Lustre: lustre-MDT0001: new disk, initializing [ 192.857932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.893974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 192.908617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.352211] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.379093] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 213.954028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.292934] Lustre: lustre-OST0000: new disk, initializing [ 214.304040] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 214.311491] Lustre: 8413:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.400911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.688124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 219.707367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 219.837263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 222.202817] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.391916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.492495] Lustre: lustre-OST0001: new disk, initializing [ 239.496252] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 239.502229] Lustre: 9468:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.559265] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 245.404533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 245.419815] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.500414] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 247.878046] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.555171] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.206349] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 278.664904] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.558793] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing yml_node [ 288.452271] Lustre: DEBUG MARKER: Client: 2.17.53.35 [ 290.884942] Lustre: DEBUG MARKER: MDS: 2.17.53.35 [ 293.398518] Lustre: DEBUG MARKER: OSS: 2.17.53.35 [ 295.739170] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jun 1 13:20:19 EDT 2026 [ 315.783349] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 317.755739] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 336.352096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 336.362161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.387191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.895525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.896830] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.918227] Lustre: Skipped 1 previous similar message [ 341.106759] Lustre: server umount lustre-MDT0000 complete [ 346.318589] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334472 with bad export cookie 10838901470338671035 [ 346.330150] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 346.334866] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 348.128956] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.130300] LustreError: 7713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.169635] Lustre: Skipped 1 previous similar message [ 348.170706] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.170711] Lustre: Skipped 3 previous similar messages [ 348.249622] LustreError: 7713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 351.136175] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.138065] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.177747] Lustre: Skipped 2 previous similar messages [ 353.164068] Lustre: server umount lustre-MDT0001 complete [ 358.864299] Lustre: server umount lustre-OST0000 complete [ 364.773636] Lustre: server umount lustre-OST0001 complete [ 372.333533] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 380.180695] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 420.665586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.097902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 421.169482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 421.349433] Lustre: lustre-MDT0000: new disk, initializing [ 421.501478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.525538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 426.750533] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 440.043381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.221315] Lustre: 14989:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 440.325673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 440.329483] Lustre: Skipped 1 previous similar message [ 440.426933] Lustre: lustre-MDT0001: new disk, initializing [ 440.540063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 440.583561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 440.587046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 445.733832] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 454.196470] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 458.661781] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 465.473230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.691857] Lustre: lustre-OST0000: new disk, initializing [ 465.695463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.700413] Lustre: 16732:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.791858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.597165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 467.620856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.706080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.193714] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 481.800581] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 488.104907] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 488.299409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 495.892401] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 496.299375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 503.776581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 503.788589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.817902] LustreError: Skipped 1 previous similar message [ 503.826236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.872119] Lustre: Skipped 2 previous similar messages [ 512.991337] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 513.085467] Lustre: server umount lustre-OST0000 complete [ 517.614548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 517.621583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.642789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 517.645167] Lustre: Skipped 3 previous similar messages [ 523.856118] Lustre: server umount lustre-MDT0000 complete [ 524.257685] LustreError: 15000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.280873] LustreError: 15000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 528.205655] LustreError: 14980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334654 with bad export cookie 10838901470338672617 [ 528.208094] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.222575] LustreError: 14980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 528.816734] Lustre: server umount lustre-MDT0001 complete [ 533.050039] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 13:24:17 (1780334657) [ 542.398860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.051577] LustreError: 18853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.168418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 548.231845] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 548.321649] LustreError: 18854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.563763] LustreError: 18853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.585714] LustreError: 18853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 558.687536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.472159] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 570.307235] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 575.033341] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 582.898432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 590.582282] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 598.287455] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 605.155904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 605.172365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.196018] Lustre: Skipped 1 previous similar message [ 605.199879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 605.215958] Lustre: Skipped 2 previous similar messages [ 607.172133] Lustre: server umount lustre-OST0000 complete [ 620.617448] Lustre: server umount lustre-MDT0000 complete [ 623.588898] LustreError: 18853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.625421] LustreError: 18853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 625.371814] LustreError: 18835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334751 with bad export cookie 10838901470338673541 [ 625.386987] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.396901] LustreError: 18835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 625.947622] Lustre: server umount lustre-MDT0001 complete [ 638.467357] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 641.646670] Key type lgssc unregistered [ 641.966078] LNet: 22031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.992344] LNetError: 22031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 642.012546] LNet: Removed LNI 192.168.201.129@tcp [ 642.885278] Key type .llcrypt unregistered [ 642.887398] Key type ._llcrypt unregistered [ 675.196285] Key type ._llcrypt registered [ 675.199376] Key type .llcrypt registered [ 675.331500] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 676.420983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 676.445641] alg: No test for adler32 (adler32-zlib) [ 677.724783] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 677.960394] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 679.599128] Key type lgssc registered [ 681.004929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 692.477269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 692.521966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.096791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 700.604348] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 704.482334] LustreError: 23428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.504737] LustreError: 23428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 709.615139] LustreError: 23429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.432506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.904748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 715.901802] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 721.902168] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 726.093954] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 736.039605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 736.707574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.746154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 745.935201] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 753.362734] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 773.921665] Lustre: server umount lustre-OST0000 complete [ 781.794065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.795892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 781.806224] Lustre: Skipped 1 previous similar message [ 781.819887] Lustre: Skipped 1 previous similar message [ 786.919845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 786.939086] Lustre: Skipped 1 previous similar message [ 788.041714] Lustre: server umount lustre-MDT0000 complete [ 792.032771] LustreError: 23429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 792.065576] LustreError: 23429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 792.246079] LustreError: 23409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334918 with bad export cookie 15567958827800892041 [ 792.257277] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 792.262486] LustreError: 23409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 802.028861] Lustre: server umount lustre-MDT0001 complete [ 815.555094] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 818.552822] Key type lgssc unregistered [ 818.829427] LNet: 27054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.838268] LNetError: 27054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.864530] LNet: Removed LNI 192.168.201.129@tcp [ 820.243933] Key type .llcrypt unregistered [ 820.245725] Key type ._llcrypt unregistered [ 847.978530] Key type ._llcrypt registered [ 847.981064] Key type .llcrypt registered [ 848.081717] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 849.276747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 849.340626] alg: No test for adler32 (adler32-zlib) [ 850.580373] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 850.797477] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 852.559177] Key type lgssc registered [ 853.782742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 862.479605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 862.523465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 864.195522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 868.836916] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 874.469528] LustreError: 28452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 874.491597] LustreError: 28452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 879.595361] LustreError: 28451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 880.391793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 881.150406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 886.370648] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 893.033080] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 897.867239] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 913.599260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 913.842639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 919.444853] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 926.440649] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 934.340938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 944.613101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 944.614819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 944.635896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 944.678888] Lustre: Skipped 1 previous similar message [ 949.538653] Lustre: server umount lustre-OST0000 complete [ 954.849973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.858184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 954.872415] Lustre: Skipped 1 previous similar message [ 954.876191] Lustre: Skipped 1 previous similar message [ 959.970241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 959.994187] Lustre: Skipped 2 previous similar messages [ 960.766167] Lustre: server umount lustre-MDT0000 complete [ 965.104407] LustreError: 29165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.145067] LustreError: 29165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 967.325922] LustreError: 28432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335093 with bad export cookie 11337347552956527498 [ 967.343773] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 967.358196] LustreError: 28432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 967.899939] Lustre: server umount lustre-MDT0001 complete [ 986.619068] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 991.125472] Key type lgssc unregistered [ 991.393661] LNet: 32015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.402904] LNetError: 32015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.418851] LNet: Removed LNI 192.168.201.129@tcp [ 992.584873] Key type .llcrypt unregistered [ 992.591937] Key type ._llcrypt unregistered [ 1020.954193] Key type ._llcrypt registered [ 1020.961618] Key type .llcrypt registered [ 1021.077317] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1022.699848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.719254] alg: No test for adler32 (adler32-zlib) [ 1023.787977] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 1024.064322] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1025.791480] Key type lgssc registered [ 1027.056833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1034.948061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1034.995314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.522839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.954440] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1045.991121] LustreError: 33413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1046.014818] LustreError: 33413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1050.377183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1050.974137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1055.112442] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1061.684974] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1067.344571] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1075.900377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1076.201695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.328208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1082.003780] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1087.623807] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1100.780374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1100.787834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.788482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1100.805658] Lustre: Skipped 1 previous similar message [ 1101.808129] Lustre: server umount lustre-OST0000 complete [ 1107.425647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1107.432597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.444417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1107.450254] Lustre: Skipped 1 previous similar message [ 1111.009774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1111.010206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.774257] Lustre: server umount lustre-MDT0000 complete [ 1116.132973] LustreError: 33410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.163334] LustreError: 33410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1117.122661] LustreError: 33395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335243 with bad export cookie 4013972148843654091 [ 1117.129104] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.133857] LustreError: 33395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1117.609067] Lustre: server umount lustre-MDT0001 complete [ 1130.679144] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 1134.713865] Key type lgssc unregistered [ 1135.157097] LNet: 36592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.164537] LNetError: 36592:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.186389] LNet: Removed LNI 192.168.201.129@tcp [ 1136.162397] Key type .llcrypt unregistered [ 1136.164890] Key type ._llcrypt unregistered [ 1162.443733] Key type ._llcrypt registered [ 1162.445813] Key type .llcrypt registered [ 1162.535346] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1163.918941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1164.160242] alg: No test for adler32 (adler32-zlib) [ 1165.456115] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 1165.909525] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1167.655189] Key type lgssc registered [ 1169.063734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1178.271504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1178.302327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.926875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.524403] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1190.368925] LustreError: 37986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.381326] LustreError: 37986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1195.489726] LustreError: 37987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1195.736033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.151137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1201.381874] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1207.913271] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1211.995575] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1218.714404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1219.002498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.219069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1226.560687] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1233.114893] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1240.195544] Lustre: lustre-OST0000: Not available for connect from 192.168.201.29@tcp (stopping) [ 1240.557093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.561465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1245.287652] Lustre: lustre-OST0000: Not available for connect from 192.168.201.29@tcp (stopping) [ 1245.298749] Lustre: Skipped 2 previous similar messages [ 1250.414467] Lustre: lustre-OST0000: Not available for connect from 192.168.201.29@tcp (stopping) [ 1250.425883] Lustre: Skipped 2 previous similar messages [ 1253.855501] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1254.077649] Lustre: server umount lustre-OST0000 complete [ 1274.834207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.844441] Lustre: Skipped 1 previous similar message [ 1274.859175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1274.876400] Lustre: Skipped 3 previous similar messages [ 1279.972078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1279.979347] Lustre: Skipped 1 previous similar message [ 1289.183359] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1289.338284] Lustre: server umount lustre-MDT0000 complete [ 1291.241799] LustreError: 37983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1291.265719] LustreError: 37983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1293.331370] LustreError: 37968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335419 with bad export cookie 14549821841469685882 [ 1293.344696] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1299.869527] Lustre: server umount lustre-MDT0001 complete [ 1311.332781] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 1313.817820] Key type lgssc unregistered [ 1314.098788] LNet: 41341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.133366] LNetError: 41341:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.173305] LNet: Removed LNI 192.168.201.129@tcp [ 1315.960751] Key type .llcrypt unregistered [ 1315.962738] Key type ._llcrypt unregistered [ 1339.613417] Key type ._llcrypt registered [ 1339.614729] Key type .llcrypt registered [ 1339.681768] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1340.941137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1340.979038] alg: No test for adler32 (adler32-zlib) [ 1342.034920] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 1342.206507] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1343.847194] Key type lgssc registered [ 1345.075234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1353.217705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1353.271065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.642091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.133319] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1364.966555] LustreError: 42739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.979350] LustreError: 42739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1366.818959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.148184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1370.716404] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1375.853594] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1379.813739] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1386.349386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1386.717261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.928398] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1397.170924] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1400.049637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1409.149526] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.29@tcp (stopping) [ 1410.529672] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1410.533071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1410.544330] Lustre: Skipped 1 previous similar message [ 1410.574894] Lustre: Skipped 2 previous similar messages [ 1410.772887] Lustre: server umount lustre-MDT0000 complete [ 1414.248809] LustreError: 42734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1414.460062] LustreError: 42718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335540 with bad export cookie 10425863582815035653 [ 1414.475185] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1415.648547] LustreError: 44891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.650492] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.686053] Lustre: Skipped 1 previous similar message [ 1415.697327] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1419.371646] LustreError: 42733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1419.401504] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.29@tcp (stopping) [ 1420.822764] Lustre: server umount lustre-MDT0001 complete [ 1449.951930] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1450.118419] Lustre: server umount lustre-OST0000 complete [ 1467.403359] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 1469.907797] Key type lgssc unregistered [ 1470.170746] LNet: 46243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.175578] LNetError: 46243:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.194943] LNet: Removed LNI 192.168.201.129@tcp [ 1471.057375] Key type .llcrypt unregistered [ 1471.062267] Key type ._llcrypt unregistered [ 1498.295348] Key type ._llcrypt registered [ 1498.298079] Key type .llcrypt registered [ 1498.424788] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1499.943037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1500.007953] alg: No test for adler32 (adler32-zlib) [ 1501.137778] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 1501.374123] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1503.040605] Key type lgssc registered [ 1504.054459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1513.132244] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1513.149644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1528.737394] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1533.855437] LustreError: 47612:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 1560.543202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.504883] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1572.975533] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1739.847677] Lustre: server umount lustre-OST0000 complete [ 1757.306679] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 1760.504614] Key type lgssc unregistered [ 1760.816982] LNet: 49158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.823931] LNetError: 49158:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.838636] LNet: Removed LNI 192.168.201.129@tcp [ 1761.789671] Key type .llcrypt unregistered [ 1761.791726] Key type ._llcrypt unregistered [ 1787.711760] Key type ._llcrypt registered [ 1787.716754] Key type .llcrypt registered [ 1787.827125] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1788.954812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1789.089974] alg: No test for adler32 (adler32-zlib) [ 1790.281282] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 1790.560332] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1792.239470] Key type lgssc registered [ 1793.536283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1802.332770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1802.351404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.678781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.905927] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1813.988307] LustreError: 50557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1814.016464] LustreError: 50557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1816.570111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.026992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1821.608722] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1828.423827] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1833.020541] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1840.923286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1841.433128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.431859] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1848.637887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1855.083225] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1858.019630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1858.027457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1858.045827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.041551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1859.045208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.048498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.076945] Lustre: Skipped 1 previous similar message [ 1863.825321] Lustre: server umount lustre-OST0000 complete [ 1868.257320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1868.265732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1868.285403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.138548] Lustre: server umount lustre-MDT0000 complete [ 1874.404243] LustreError: 50558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1874.427099] LustreError: 50558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1877.424061] LustreError: 50540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336003 with bad export cookie 1050496374932666941 [ 1877.437471] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.726515] Lustre: server umount lustre-MDT0001 complete [ 1888.513927] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 1891.766677] Key type lgssc unregistered [ 1892.172690] LNet: 53735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1892.197489] LNetError: 53735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.223285] LNet: Removed LNI 192.168.201.129@tcp [ 1894.103981] Key type .llcrypt unregistered [ 1894.109271] Key type ._llcrypt unregistered [ 1920.277431] Key type ._llcrypt registered [ 1920.280081] Key type .llcrypt registered [ 1920.411827] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1921.403329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1921.443016] alg: No test for adler32 (adler32-zlib) [ 1922.845944] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 1923.178477] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1924.895197] Key type lgssc registered [ 1926.335781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1934.475497] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1934.494295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1950.239402] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1955.359269] LustreError: 55105:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 1982.048014] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.699463] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1995.162335] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2002.915738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2003.542071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.629514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2008.153646] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2008.546250] LustreError: 55130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.558169] LustreError: 55130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2013.680867] LustreError: 55131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.745574] LustreError: 56029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.772606] LustreError: 56029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2019.579572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.273355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2024.729495] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2030.776434] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2034.489215] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2039.264571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2039.273290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.295440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2040.800198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2040.809698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2040.817741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.845204] Lustre: Skipped 1 previous similar message [ 2043.300444] Lustre: server umount lustre-OST0000 complete [ 2071.521499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.524128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.554952] Lustre: Skipped 1 previous similar message [ 2071.561274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2073.293839] Lustre: server umount lustre-MDT0000 complete [ 2076.660840] LustreError: 56034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.673014] LustreError: 56034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2077.379157] LustreError: 55112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336203 with bad export cookie 4549690335073976777 [ 2077.384389] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2077.392714] LustreError: 55112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2083.811969] Lustre: server umount lustre-MDT0001 complete [ 2095.055701] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2098.094722] Key type lgssc unregistered [ 2098.335947] LNet: 58495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.347621] LNetError: 58495:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2098.365346] LNet: Removed LNI 192.168.201.129@tcp [ 2099.074510] Key type .llcrypt unregistered [ 2099.075869] Key type ._llcrypt unregistered [ 2124.503630] Key type ._llcrypt registered [ 2124.506795] Key type .llcrypt registered [ 2124.586329] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2125.422144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2125.541929] alg: No test for adler32 (adler32-zlib) [ 2126.674338] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 2126.865055] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2128.529576] Key type lgssc registered [ 2129.700665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2137.680169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2137.713175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2139.407829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.390801] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2148.839171] LustreError: 59897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.871024] LustreError: 59897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2151.655016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.980082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2156.277933] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2162.867074] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2167.009525] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2173.383520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2173.765775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2180.410397] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2183.987883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2185.820097] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2204.648156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2204.660613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2204.679865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2204.684070] Lustre: Skipped 1 previous similar message [ 2205.303959] Lustre: server umount lustre-OST0000 complete [ 2209.763867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.769610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2209.786495] Lustre: Skipped 1 previous similar message [ 2209.798882] Lustre: Skipped 2 previous similar messages [ 2214.881149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2214.888098] Lustre: Skipped 1 previous similar message [ 2215.333689] Lustre: server umount lustre-MDT0000 complete [ 2219.633841] LustreError: 61515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336346 with bad export cookie 14444391559640371298 [ 2219.641791] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2219.649462] LustreError: 61515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2219.974308] Lustre: server umount lustre-MDT0001 complete [ 2232.360551] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2234.870400] Key type lgssc unregistered [ 2235.106644] LNet: 63097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2235.115147] LNetError: 63097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2236.134626] LNet: Removed LNI 192.168.201.129@tcp [ 2236.915641] Key type .llcrypt unregistered [ 2236.921055] Key type ._llcrypt unregistered [ 2264.543658] Key type ._llcrypt registered [ 2264.545658] Key type .llcrypt registered [ 2264.645512] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2266.006678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2266.037256] alg: No test for adler32 (adler32-zlib) [ 2267.168149] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 2267.362582] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2268.999166] Key type lgssc registered [ 2269.953627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2278.951471] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 13:53:23 (1780336403) [ 2284.700346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2284.718508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.196672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2290.521307] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2295.809584] LustreError: 64785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.823117] LustreError: 64785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2298.957057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2299.231187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2303.640578] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2310.223541] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2315.011158] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2322.376643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2322.997568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.713416] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2331.253547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2336.017328] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2342.290884] Lustre: Failing over lustre-MDT0000 [ 2342.671548] Lustre: server umount lustre-MDT0000 complete [ 2344.071437] LustreError: 64780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.451447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2345.456873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.465177] LustreError: 64785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.992714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.018149] Lustre: Skipped 1 previous similar message [ 2349.160944] LustreError: 64780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.182974] LustreError: 64780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2352.753382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2352.869532] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2353.054946] Lustre: *** cfs_fail_loc=135, val=0*** [ 2353.162972] LustreError: 67535:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2353.171373] LustreError: 67535:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.129@tcp: cfg command failed: rc = -2 [ 2353.185501] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2353.185501] [ 2353.193472] LustreError: MGC192.168.201.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2353.206090] LustreError: 67523:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2353.217041] LustreError: 67523:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -2 [ 2353.232780] LustreError: 67523:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2353.240768] LustreError: 67523:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2353.282599] Lustre: server umount lustre-MDT0000 complete [ 2353.286996] LustreError: 67523:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2354.279053] LustreError: 64780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.293486] LustreError: 64780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2360.450731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.555527] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2360.783606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.840122] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2364.390787] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2365.077800] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2365.947951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2365.986081] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2366.052332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2375.139416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2375.146723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.163810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2376.172691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2376.191939] Lustre: Skipped 1 previous similar message [ 2380.053519] Lustre: server umount lustre-OST0000 complete [ 2386.400209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2386.404497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.426083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2386.464560] Lustre: Skipped 2 previous similar messages [ 2390.506434] Lustre: server umount lustre-MDT0000 complete [ 2391.530334] LustreError: 64786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.552026] LustreError: 64786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2394.296810] LustreError: 68547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336520 with bad export cookie 7022255711233489753 [ 2394.305558] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2394.316475] LustreError: 68547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2394.957585] Lustre: server umount lustre-MDT0001 complete [ 2405.048410] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2408.589910] Key type lgssc unregistered [ 2409.148793] LNet: 69283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2409.154285] LNetError: 69283:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2409.163781] LNet: Removed LNI 192.168.201.129@tcp [ 2409.969362] Key type .llcrypt unregistered [ 2409.987605] Key type ._llcrypt unregistered [ 2441.388366] Key type ._llcrypt registered [ 2441.392479] Key type .llcrypt registered [ 2441.556777] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2443.424819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2443.446344] alg: No test for adler32 (adler32-zlib) [ 2444.587800] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 2444.768992] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2446.463186] Key type lgssc registered [ 2448.643107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2456.891292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2456.908513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.279596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2461.965459] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2467.814971] LustreError: 70680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2467.835157] LustreError: 70680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2470.147637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.615150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2474.970356] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2480.430606] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2484.560418] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2490.909192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2491.190138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.203831] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2503.442101] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2503.487462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2508.519929] Lustre: Failing over lustre-MDT0000 [ 2508.782072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.796198] LustreError: 70680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.809883] LustreError: 70680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2508.874324] Lustre: server umount lustre-MDT0000 complete [ 2509.955236] LustreError: 70675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.970017] LustreError: 70675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2513.890338] LustreError: 70681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.901103] LustreError: 70681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2518.033973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.206596] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2518.364865] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2518.524066] LustreError: 73427:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2518.532519] LustreError: 73427:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.129@tcp: cfg command failed: rc = -9 [ 2518.541514] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2518.541514] [ 2518.559123] LustreError: MGC192.168.201.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2518.578338] LustreError: 73416:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2518.589050] LustreError: 73416:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -9 [ 2518.605176] LustreError: 73416:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2518.611654] LustreError: 73416:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2518.640535] Lustre: server umount lustre-MDT0000 complete [ 2518.642971] LustreError: 73416:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2519.009673] LustreError: 70675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.019735] LustreError: 70675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2525.036768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2525.120733] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2525.381212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2525.423204] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2529.007840] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2530.414648] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2530.788616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2530.830518] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2530.861937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2535.908828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.912490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2535.921041] Lustre: Skipped 3 previous similar messages [ 2537.723838] Lustre: server umount lustre-OST0000 complete [ 2546.149791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.161594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2546.166416] Lustre: Skipped 1 previous similar message [ 2546.167679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2546.172527] Lustre: Skipped 2 previous similar messages [ 2547.525590] Lustre: server umount lustre-MDT0000 complete [ 2550.795101] LustreError: 70659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336677 with bad export cookie 14309533209469532086 [ 2550.800386] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2550.805621] LustreError: 70659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2551.168537] Lustre: server umount lustre-MDT0001 complete [ 2563.570836] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2566.370151] Key type lgssc unregistered [ 2566.609092] LNet: 75176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2566.618286] LNetError: 75176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2566.635364] LNet: Removed LNI 192.168.201.129@tcp [ 2567.370696] Key type .llcrypt unregistered [ 2567.372847] Key type ._llcrypt unregistered [ 2583.071672] Key type ._llcrypt registered [ 2583.074114] Key type .llcrypt registered [ 2583.186392] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2583.543527] Key type .llcrypt unregistered [ 2583.546629] Key type ._llcrypt unregistered [ 2608.915492] Key type ._llcrypt registered [ 2608.918266] Key type .llcrypt registered [ 2608.993537] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2610.188742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2610.250521] alg: No test for adler32 (adler32-zlib) [ 2611.388987] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 2611.624551] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2613.296211] Key type lgssc registered [ 2614.765881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2623.871676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2623.895155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.610218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2631.230798] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2635.744126] LustreError: 77046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.758208] LustreError: 77046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2640.875932] LustreError: 77047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.622348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.881780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2646.613970] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2652.496959] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2656.656663] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2664.437079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2664.857978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2671.032520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2671.416962] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2676.731492] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2686.439046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2686.450540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2686.474129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2689.803421] Lustre: server umount lustre-OST0000 complete [ 2696.673602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.681765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2696.695869] Lustre: Skipped 2 previous similar messages [ 2696.699185] Lustre: Skipped 1 previous similar message [ 2700.906400] Lustre: server umount lustre-MDT0000 complete [ 2701.796327] LustreError: 78018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.405979] LustreError: 77026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336830 with bad export cookie 4046369710490373885 [ 2704.422956] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.424929] LustreError: 77026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2704.927746] Lustre: server umount lustre-MDT0001 complete [ 2715.497185] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2718.179907] Key type lgssc unregistered [ 2718.480974] LNet: 80225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2718.484753] LNetError: 80225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2718.498636] LNet: Removed LNI 192.168.201.129@tcp [ 2719.386353] Key type .llcrypt unregistered [ 2719.387953] Key type ._llcrypt unregistered [ 2752.240873] Key type ._llcrypt registered [ 2752.248442] Key type .llcrypt registered [ 2752.395239] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2753.613956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2753.685874] alg: No test for adler32 (adler32-zlib) [ 2754.661338] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 2754.935391] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2756.607199] Key type lgssc registered [ 2757.540677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2763.576876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2763.612447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.172897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.429546] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2775.537310] LustreError: 81635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.566742] LustreError: 81635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2777.612647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2777.877971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2781.911279] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2787.326178] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2791.009506] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2798.751893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2799.072077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.896135] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2811.377013] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2811.422233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2818.537521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2818.547170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.562894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2821.600725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2821.605167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2821.611142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.636899] Lustre: Skipped 1 previous similar message [ 2824.641629] Lustre: server umount lustre-OST0000 complete [ 2829.293044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2829.297072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.304404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2831.842416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2831.844104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2834.547500] Lustre: server umount lustre-MDT0000 complete [ 2836.982503] LustreError: 83874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.010250] LustreError: 83874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2838.922848] LustreError: 81616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336965 with bad export cookie 1718280654354418631 [ 2838.929894] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2838.933313] LustreError: 81616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2839.198711] Lustre: server umount lustre-MDT0001 complete [ 2851.587213] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2854.413455] Key type lgssc unregistered [ 2854.679826] LNet: 84812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2854.686552] LNetError: 84812:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2854.697615] LNet: Removed LNI 192.168.201.129@tcp [ 2855.597127] Key type .llcrypt unregistered [ 2855.600513] Key type ._llcrypt unregistered [ 2876.351994] Key type ._llcrypt registered [ 2876.354528] Key type .llcrypt registered [ 2876.472382] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2877.479531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2877.495113] alg: No test for adler32 (adler32-zlib) [ 2878.579579] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 2878.786498] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2880.463193] Key type lgssc registered [ 2881.614738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2889.167116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2889.201640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.705297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2895.027401] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2900.964150] LustreError: 86189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.979487] LustreError: 86189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2902.472956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.732194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2906.503679] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2911.886727] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2915.630747] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2923.479179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2923.827886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2931.001614] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2936.126499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2937.960237] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2951.651181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.656612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2951.674047] Lustre: Skipped 1 previous similar message [ 2951.674952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2951.689703] LustreError: Skipped 1 previous similar message [ 2953.379743] Lustre: server umount lustre-OST0000 complete [ 2959.328783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2959.339313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.352891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2959.356372] Lustre: Skipped 1 previous similar message [ 2961.888435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.888881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2961.905543] Lustre: Skipped 1 previous similar message [ 2963.125721] Lustre: server umount lustre-MDT0000 complete [ 2967.009489] LustreError: 86900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.044509] LustreError: 86900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2967.589188] LustreError: 86170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337094 with bad export cookie 2081338103523824913 [ 2967.598261] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2967.600225] LustreError: 86170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2968.084104] Lustre: server umount lustre-MDT0001 complete [ 2985.157257] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2988.169453] Key type lgssc unregistered [ 2988.548489] LNet: 89414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2988.556046] LNetError: 89414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2989.608087] LNet: Removed LNI 192.168.201.129@tcp [ 2990.433263] Key type .llcrypt unregistered [ 2990.434958] Key type ._llcrypt unregistered [ 3012.978475] Key type ._llcrypt registered [ 3012.980510] Key type .llcrypt registered [ 3013.095320] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3013.968531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3014.065300] alg: No test for adler32 (adler32-zlib) [ 3015.070147] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 3015.236972] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 3016.935197] Key type lgssc registered [ 3018.024183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3026.016900] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3026.057141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3041.760588] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3046.879776] LustreError: 90783:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 3073.696333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3079.159399] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3084.790924] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3090.823810] Lustre: server umount lustre-OST0000 complete [ 3099.853980] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 14:07:04 (1780337224) [ 3105.912424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.457486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.987350] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3111.908561] LustreError: 92308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3111.925163] LustreError: 92308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3116.008841] LustreError: 92309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.127469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.583304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3123.632175] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3129.655117] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3133.634746] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3140.098471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3140.484693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3146.020707] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3150.813747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3151.357351] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3164.128325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3164.142209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3164.176531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3166.182520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3166.187093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3166.194924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.223078] Lustre: Skipped 1 previous similar message [ 3167.746357] Lustre: server umount lustre-OST0000 complete [ 3175.905154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3175.911517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.931182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3177.843932] Lustre: server umount lustre-MDT0000 complete [ 3181.539977] LustreError: 94409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3181.556854] LustreError: 94409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3181.648353] LustreError: 92290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337308 with bad export cookie 4089737910331782894 [ 3181.651593] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3181.662499] LustreError: 92290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3182.120076] Lustre: server umount lustre-MDT0001 complete [ 3193.280705] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 3196.119158] Key type lgssc unregistered [ 3196.420190] LNet: 95646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3196.437680] LNetError: 95646:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3196.461879] LNet: Removed LNI 192.168.201.129@tcp [ 3197.229593] Key type .llcrypt unregistered [ 3197.232742] Key type ._llcrypt unregistered [ 3239.866759] Key type ._llcrypt registered [ 3239.868885] Key type .llcrypt registered [ 3239.959350] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3240.724577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3240.829469] alg: No test for adler32 (adler32-zlib) [ 3241.982460] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 3242.251264] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 3243.903543] Key type lgssc registered [ 3244.717825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3252.181212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3252.197601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.826324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3258.342441] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3263.971357] LustreError: 97816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3263.983940] LustreError: 97816:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3267.258927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.637434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3271.810199] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3277.423462] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3281.249791] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3288.144775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3288.515353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3294.764766] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3298.734468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3300.909420] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3313.120218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3313.123442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.144491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3314.145613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3314.149318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3314.167472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.567918] Lustre: server umount lustre-OST0000 complete [ 3323.872285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3323.881386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3323.895294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3323.903667] Lustre: Skipped 1 previous similar message [ 3326.638783] Lustre: server umount lustre-MDT0000 complete [ 3329.505382] LustreError: 97813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.545898] LustreError: 97813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3330.484461] LustreError: 97798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337456 with bad export cookie 16588483808320644619 [ 3330.487691] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3330.495487] LustreError: 97798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3330.961030] Lustre: server umount lustre-MDT0001 complete [ 3341.938644] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 3344.534871] Key type lgssc unregistered [ 3344.873322] LNet: 100994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3344.881815] LNetError: 100994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3345.960754] LNet: Removed LNI 192.168.201.129@tcp [ 3346.806283] Key type .llcrypt unregistered [ 3346.808470] Key type ._llcrypt unregistered [ 3366.286411] Key type ._llcrypt registered [ 3366.288129] Key type .llcrypt registered [ 3366.389817] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3367.674016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3367.815403] alg: No test for adler32 (adler32-zlib) [ 3368.849978] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 3369.083474] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 3370.784390] Key type lgssc registered [ 3372.266479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3380.264310] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3380.304273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3395.871491] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3400.991473] LustreError: 102221:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 3427.743186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.998736] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3439.627103] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3447.062029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3447.246926] LustreError: MGC192.168.201.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3447.254369] LustreError: 103132:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3447.259660] LustreError: 103132:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -2 [ 3447.267145] LustreError: 103132:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3447.273887] LustreError: 102227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337573 with bad export cookie 7292701483623879817 [ 3447.285632] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3447.357787] Lustre: server umount lustre-MDT0000 complete [ 3447.362099] LustreError: 103132:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3475.585466] Lustre: server umount lustre-OST0000 complete [ 3484.493434] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 3495.161415] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3535.974918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3536.554839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3536.638524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3536.851726] Lustre: lustre-MDT0000: new disk, initializing [ 3536.997059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3537.014277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3541.741819] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3553.411330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3553.540734] Lustre: 105891:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3553.590423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3553.596792] Lustre: Skipped 1 previous similar message [ 3553.670441] Lustre: lustre-MDT0001: new disk, initializing [ 3553.776136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3553.809257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3553.824306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3558.101351] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3566.066660] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3570.295807] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3577.960599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3578.338479] Lustre: lustre-OST0000: new disk, initializing [ 3578.343352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3578.356703] Lustre: 107636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3578.483609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3584.376726] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3587.714271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3587.727851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3587.947186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3592.550156] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3597.009455] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3597.233810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3601.373787] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3601.671231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3608.547811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3608.551982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.563959] LustreError: Skipped 1 previous similar message [ 3608.582088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3608.596328] Lustre: Skipped 1 previous similar message [ 3610.020660] Lustre: server umount lustre-OST0000 complete [ 3615.203989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3615.223150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.249552] Lustre: Skipped 1 previous similar message [ 3615.264236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3618.784294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3618.789116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.645263] Lustre: server umount lustre-MDT0000 complete [ 3625.018961] LustreError: 105881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337751 with bad export cookie 7292701483623879922 [ 3625.025580] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3625.034500] LustreError: 105881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3625.456379] Lustre: server umount lustre-MDT0001 complete [ 3635.126948] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 14:15:59 (1780337759) [ 3639.180477] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3657.349947] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 3666.852901] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3706.709783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.225745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3707.294760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3707.452295] Lustre: lustre-MDT0000: new disk, initializing [ 3707.650686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3707.687212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3712.668676] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3724.857703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.942267] Lustre: 112357:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3724.981996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3724.985463] Lustre: Skipped 1 previous similar message [ 3725.035418] Lustre: lustre-MDT0001: new disk, initializing [ 3725.113051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3725.166906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3725.184683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3729.550637] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3737.397296] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3741.445944] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3747.746943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3748.166718] Lustre: lustre-OST0000: new disk, initializing [ 3748.171729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3748.192246] Lustre: 114099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3748.337950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3753.857613] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3757.161125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3757.169274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3757.282469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3761.904041] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3766.867938] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3767.040082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3771.570689] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3771.773209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3778.016352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3778.023522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3778.027170] LustreError: Skipped 1 previous similar message [ 3778.032680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3778.063426] Lustre: Skipped 1 previous similar message [ 3778.088157] Lustre: Skipped 2 previous similar messages [ 3779.593561] Lustre: server umount lustre-OST0000 complete [ 3786.728463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3786.733966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.745020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3789.862595] Lustre: server umount lustre-MDT0000 complete [ 3793.385625] LustreError: 112364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3793.403302] LustreError: 112364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3793.441402] LustreError: 114098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337919 with bad export cookie 7292701483623880846 [ 3793.443652] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3793.461247] LustreError: 114098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3793.893629] Lustre: server umount lustre-MDT0001 complete [ 3802.103901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3802.405277] LustreError: 116168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3802.468889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.938600] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3807.714181] LustreError: 116169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.810682] LustreError: 116168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3815.036791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.944559] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3826.141373] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3830.816880] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3838.081166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3844.895589] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3850.235561] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3858.291323] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3861.990483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3862.000823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3862.002048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3862.002054] Lustre: Skipped 2 previous similar messages [ 3862.008533] Lustre: Skipped 2 previous similar messages [ 3867.913767] Lustre: server umount lustre-OST0000 complete [ 3871.712546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3877.351091] LustreError: 116169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3877.365097] LustreError: 116169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3877.415625] Lustre: server umount lustre-MDT0000 complete [ 3881.249916] LustreError: 116148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338007 with bad export cookie 7292701483623881770 [ 3881.254430] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3881.266704] LustreError: 116148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3881.574949] Lustre: server umount lustre-MDT0001 complete [ 3892.438897] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 3894.787213] Key type lgssc unregistered [ 3895.092355] LNet: 119442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3895.101812] LNetError: 119442:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3895.114543] LNet: Removed LNI 192.168.201.129@tcp [ 3895.991267] Key type .llcrypt unregistered [ 3895.993805] Key type ._llcrypt unregistered [ 3912.453665] Key type ._llcrypt registered [ 3912.459212] Key type .llcrypt registered [ 3912.557253] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 3924.829783] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3925.931215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3925.963255] alg: No test for adler32 (adler32-zlib) [ 3927.091304] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 3927.304813] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 3928.975497] Key type lgssc registered [ 3929.994778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3963.210422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3963.285788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.720226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3964.778270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3964.869638] Lustre: lustre-MDT0000: new disk, initializing [ 3964.959134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.996383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3969.084073] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3978.390946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3978.492786] Lustre: 122552:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3978.546503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3978.551122] Lustre: Skipped 1 previous similar message [ 3978.629329] Lustre: lustre-MDT0001: new disk, initializing [ 3978.716697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3978.764166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3978.780426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3982.390340] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3988.905820] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3993.048871] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3999.957785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4000.169615] Lustre: lustre-OST0000: new disk, initializing [ 4000.172955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4000.181050] Lustre: 124297:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4000.263292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4006.489724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4006.505882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4006.639571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4007.761711] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4016.765733] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4020.772697] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4021.078163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4025.140799] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4025.365776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4027.360243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4027.366792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4027.373825] LustreError: Skipped 1 previous similar message [ 4027.390561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4027.399567] Lustre: Skipped 1 previous similar message [ 4030.456941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4030.467333] Lustre: Skipped 1 previous similar message [ 4033.463483] Lustre: server umount lustre-OST0000 complete [ 4040.160685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4040.166551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.214065] Lustre: Skipped 1 previous similar message [ 4040.226282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4043.645681] Lustre: server umount lustre-MDT0000 complete [ 4048.240745] LustreError: 122544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338174 with bad export cookie 3679594474240281108 [ 4048.243164] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4048.255423] LustreError: 122544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4048.876647] Lustre: server umount lustre-MDT0001 complete [ 4056.163185] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 14:23:00 (1780338180) [ 4064.040679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.547934] LustreError: 126560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.577322] LustreError: 126560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4064.676923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.616967] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4069.859932] LustreError: 126561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.985276] LustreError: 126560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4079.337494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4079.700991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4083.732693] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4088.908783] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4092.212204] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4094.946645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4094.947443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4094.956675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4094.956684] Lustre: Skipped 1 previous similar message [ 4095.005864] Lustre: Skipped 2 previous similar messages [ 4100.075490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4100.085585] Lustre: Skipped 2 previous similar messages [ 4108.255582] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4108.405603] Lustre: server umount lustre-MDT0000 complete [ 4110.304306] LustreError: 126560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.346421] LustreError: 126560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4111.593574] LustreError: 126542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338238 with bad export cookie 3679594474240282032 [ 4111.595289] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4111.610510] LustreError: 126542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4111.966360] Lustre: server umount lustre-MDT0001 complete [ 4121.133905] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 14:24:05 (1780338245) [ 4127.708626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4143.391769] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4148.511872] LustreError: 128750:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 4174.240893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4180.362348] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4186.239838] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4188.333957] Lustre: server umount lustre-OST0000 complete [ 4196.665788] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 14:25:20 (1780338320) [ 4202.954445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4203.539401] LustreError: 130082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.614663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.666735] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4213.730097] LustreError: 130083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4213.742773] LustreError: 130083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4216.363477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.045351] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4226.340964] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4229.802334] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4236.402377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4236.901120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4236.910134] Lustre: Skipped 1 previous similar message [ 4242.812631] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4249.291823] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4257.763513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4257.775068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.803707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4257.808137] Lustre: Skipped 2 previous similar messages [ 4262.754404] Lustre: server umount lustre-MDT0000 complete [ 4264.934862] LustreError: 131969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.952581] LustreError: 131969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4266.438927] LustreError: 130064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338392 with bad export cookie 3679594474240282697 [ 4266.456068] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4266.459795] LustreError: 130064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4266.711863] Lustre: server umount lustre-MDT0001 complete [ 4279.852803] Lustre: server umount lustre-OST0000 complete [ 4288.310118] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 14:26:52 (1780338412) [ 4295.256114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.813764] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4301.280613] LustreError: 133397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.294721] LustreError: 133397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4307.212837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.530818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4307.544379] Lustre: Skipped 1 previous similar message [ 4311.887469] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4318.204652] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4322.187695] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4329.326386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4335.900765] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4340.935696] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4340.978730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4345.608380] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4345.814986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4349.549411] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4349.706911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4351.456729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4351.461777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4351.465937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4351.471655] Lustre: Skipped 3 previous similar messages [ 4351.479336] Lustre: Skipped 4 previous similar messages [ 4357.646250] Lustre: server umount lustre-OST0000 complete [ 4361.700046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4361.713137] Lustre: Skipped 1 previous similar message [ 4367.324881] Lustre: server umount lustre-MDT0000 complete [ 4370.749828] LustreError: 133378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338497 with bad export cookie 3679594474240283768 [ 4370.756542] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4370.761631] LustreError: 133378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4370.989633] Lustre: server umount lustre-MDT0001 complete [ 4379.464598] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 14:28:23 (1780338503) [ 4386.015714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4401.632112] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4406.752143] LustreError: 137183:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 4438.381891] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4443.445733] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4450.218304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.641112] LustreError: 137208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.655612] LustreError: 137208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4450.713518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4450.723270] Lustre: Skipped 2 previous similar messages [ 4450.815725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4455.354227] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4464.345192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4469.814763] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4475.496411] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4479.111571] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4483.503792] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4483.674879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4488.687837] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4488.966871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4492.783065] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4492.785086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4492.798267] Lustre: Skipped 1 previous similar message [ 4492.803395] Lustre: Skipped 6 previous similar messages [ 4497.374544] Lustre: server umount lustre-MDT0000 complete [ 4500.628276] LustreError: 137190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338627 with bad export cookie 3679594474240284790 [ 4500.634114] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4500.637136] LustreError: 137190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4500.937512] Lustre: server umount lustre-MDT0001 complete [ 4513.737489] Lustre: server umount lustre-OST0000 complete [ 4523.211588] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 14:30:47 (1780338647) [ 4531.981698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4547.679539] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4552.799526] LustreError: 141006:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 4583.871822] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4589.007226] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4596.091431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.483702] LustreError: 141031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.506799] LustreError: 141031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4596.659621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4600.457544] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4608.978748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4615.537534] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4622.594545] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4626.529305] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4632.996805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4633.306972] Lustre: lustre-OST0001: new disk, initializing [ 4633.311193] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4633.322392] Lustre: 143539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4639.802632] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4641.865482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4641.878338] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4641.955059] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4647.016161] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4651.741533] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4652.005761] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4656.707054] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4656.913666] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4660.704244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4660.724234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4660.754091] Lustre: Skipped 1 previous similar message [ 4660.763203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4660.766130] Lustre: Skipped 2 previous similar messages [ 4664.874761] Lustre: server umount lustre-OST0000 complete [ 4675.176859] Lustre: server umount lustre-OST0001 complete [ 4681.186528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4681.195365] LustreError: Skipped 2 previous similar messages [ 4685.018079] Lustre: server umount lustre-MDT0000 complete [ 4688.501733] LustreError: 143538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338814 with bad export cookie 3679594474240285742 [ 4688.505808] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4688.512455] LustreError: 143538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4688.897590] Lustre: server umount lustre-MDT0001 complete [ 4712.771474] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 14:33:57 (1780338837) [ 4714.383961] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4716.481321] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4718.561535] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 14:34:02 (1780338842) [ 4725.642700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.924738] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4725.978767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4726.293377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4726.298415] Lustre: Skipped 5 previous similar messages [ 4730.310744] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4739.615120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4739.727078] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4739.771314] Lustre: 146950:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4743.922754] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4749.814844] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4754.401402] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4763.088628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4763.395839] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4769.548353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4772.071846] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4777.973302] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4782.953158] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4783.147542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4789.834759] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4790.084533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4795.360647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4795.370761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.374035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4795.402633] Lustre: Skipped 6 previous similar messages [ 4795.429857] Lustre: Skipped 8 previous similar messages [ 4798.662171] Lustre: server umount lustre-OST0000 complete [ 4822.901552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4830.201541] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4834.724282] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4834.755235] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4834.756581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4837.151420] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4841.682596] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4841.928515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4846.645607] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4846.941635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4851.249091] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c9f02f03800.ost_server_uuid 50 [ 4853.019640] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c9f02f03800.ost_server_uuid in FULL state after 0 sec [ 4853.265370] LustreError: 148135:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.29@tcp) returned error from blocking AST (req@000000001e56729b x1866819754523648 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000a82ac65e/0x33108badd2952588 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.29@tcp remote: 0xfbee1422463b56a7 expref: 5 pid: 148799 timeout: 4953 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4853.300327] LustreError: lustre-MDT0000: A client on nid 192.168.201.29@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4853.308132] LustreError: 146947:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.29@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000a82ac65e/0x33108badd2952588 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.29@tcp remote: 0xfbee1422463b56a7 expref: 6 pid: 148799 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4860.399566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4860.420405] LustreError: Skipped 2 previous similar messages [ 4863.809368] Lustre: server umount lustre-OST0000 complete [ 4874.714816] Lustre: server umount lustre-MDT0000 complete [ 4875.745245] LustreError: 146959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4875.758981] LustreError: 146959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4879.132705] LustreError: 146944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339005 with bad export cookie 3679594474240286925 [ 4879.145749] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4879.155245] LustreError: 146944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4885.731452] Lustre: server umount lustre-MDT0001 complete [ 4896.581263] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 4899.402334] Key type lgssc unregistered [ 4899.754684] LNet: 152430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4899.769387] LNetError: 152430:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4899.793180] LNet: Removed LNI 192.168.201.129@tcp [ 4900.948347] Key type .llcrypt unregistered [ 4900.951925] Key type ._llcrypt unregistered [ 4928.044979] Key type ._llcrypt registered [ 4928.047209] Key type .llcrypt registered [ 4928.241348] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 4929.742577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4929.911299] alg: No test for adler32 (adler32-zlib) [ 4931.084820] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 4931.402837] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 4933.127194] Key type lgssc registered [ 4934.393488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4942.556693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4942.576138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4944.312439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4949.267341] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4953.571076] LustreError: 153832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4953.589314] LustreError: 153832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4958.251552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4958.689545] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4958.882213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4964.953571] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4971.024652] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4975.067140] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4982.471715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4982.728564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4988.229837] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4988.978810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4993.576975] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5000.964441] Lustre: Failing over lustre-MDT0000 [ 5001.193493] Lustre: server umount lustre-MDT0000 complete [ 5001.372102] LustreError: 153828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5004.262851] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5004.270461] LustreError: 153831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5004.293136] Lustre: Skipped 2 previous similar messages [ 5004.327403] LustreError: 153831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5009.382406] LustreError: 153826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5009.396356] LustreError: 153826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5013.907348] LustreError: 153812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.29@tcp arrived at 1780339140 with bad export cookie 12295921169390019615 [ 5013.930892] LustreError: 153812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5014.496850] LustreError: 153832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5014.507036] LustreError: 153832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5019.615337] Lustre: 153143:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780339130/real 1780339130] req@00000000fdfae6e1 x1866820807058304/t0(0) o400->MGC192.168.201.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780339146 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5019.648374] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5021.654840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5024.740626] LustreError: 153827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5024.761097] LustreError: 153827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5029.857400] LustreError: 153142:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009a137e0e x1866820807063296/t0(0) o250->MGC192.168.201.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5030.218029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5030.272558] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5034.624284] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5035.496524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5035.507635] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5037.248473] Lustre: lustre-MDT0000: Denying connection for new client 05bbc079-c727-42b0-a5cd-891651f47980 (at 192.168.201.29@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5045.732825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5045.744590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5050.336138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5050.344144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5050.366463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5051.752412] Lustre: server umount lustre-OST0000 complete [ 5055.750294] LustreError: 157209:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5055.754839] Lustre: 156573:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5055.760936] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5055.772443] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5055.782769] LustreError: 156573:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004de0e578 x1866820807077632/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5055.794590] LustreError: 156573:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5055.800623] LustreError: 156573:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5055.807132] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5055.983824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5056.035632] LustreError: 156921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5056.049918] LustreError: 156921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5056.185832] Lustre: server umount lustre-MDT0000 complete [ 5060.034343] LustreError: 153813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339186 with bad export cookie 12295921169390019713 [ 5060.058067] LustreError: 153813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5060.063342] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5062.633877] Lustre: server umount lustre-MDT0001 complete [ 5072.906700] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 5075.954806] Key type lgssc unregistered [ 5076.192887] LNet: 157946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5076.197468] LNetError: 157946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5077.222358] LNet: Removed LNI 192.168.201.129@tcp [ 5077.932610] Key type .llcrypt unregistered [ 5077.934498] Key type ._llcrypt unregistered [ 5103.143753] Key type ._llcrypt registered [ 5103.146899] Key type .llcrypt registered [ 5103.231320] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 5104.202256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5104.244560] alg: No test for adler32 (adler32-zlib) [ 5105.449680] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 5105.654896] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 5107.443425] Key type lgssc registered [ 5108.678304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5117.668779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5117.686425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5119.055315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.822506] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5129.184842] LustreError: 159345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5129.205585] LustreError: 159345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5132.570664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5133.047583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5137.446408] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5143.800137] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5147.535531] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5154.475909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5154.721355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5160.945120] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5163.962515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5166.886234] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5174.251920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5174.255828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5174.276203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5174.320952] Lustre: Skipped 1 previous similar message [ 5178.322550] Lustre: server umount lustre-OST0000 complete [ 5184.480325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5184.484312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.504070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5184.518218] Lustre: Skipped 1 previous similar message [ 5184.536982] Lustre: Skipped 1 previous similar message [ 5188.640667] Lustre: server umount lustre-MDT0000 complete [ 5189.617659] LustreError: 161188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5189.630180] LustreError: 161188:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5192.828833] LustreError: 159325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339319 with bad export cookie 4063185143505672383 [ 5192.833503] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5192.852579] LustreError: 159325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5193.169870] Lustre: server umount lustre-MDT0001 complete [ 5209.862914] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 5214.553692] Key type lgssc unregistered [ 5215.040319] LNet: 162523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5215.052019] LNetError: 162523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5215.081231] LNet: Removed LNI 192.168.201.129@tcp [ 5216.360675] Key type .llcrypt unregistered [ 5216.362991] Key type ._llcrypt unregistered [ 5253.188873] Key type ._llcrypt registered [ 5253.191614] Key type .llcrypt registered [ 5253.323915] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 5254.825949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5254.853226] alg: No test for adler32 (adler32-zlib) [ 5255.908768] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 5256.191686] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 5257.951178] Key type lgssc registered [ 5259.381816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5267.837754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5267.854947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5269.379071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5273.782953] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5278.700802] LustreError: 164211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5278.725053] LustreError: 164211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5283.047308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5283.378637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5287.412692] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5292.520752] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5296.257503] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5304.125797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5304.425732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5310.315903] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5313.651324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5315.556558] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5324.258705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5324.270068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5324.273187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5324.298095] Lustre: Skipped 1 previous similar message [ 5324.317529] Lustre: Skipped 1 previous similar message [ 5327.649497] Lustre: server umount lustre-OST0000 complete [ 5334.504193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5334.512686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5334.514319] LustreError: Skipped 1 previous similar message [ 5334.516115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5334.559612] Lustre: Skipped 1 previous similar message [ 5337.644199] Lustre: server umount lustre-MDT0000 complete [ 5339.617746] LustreError: 164208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5339.635892] LustreError: 164208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5341.442534] LustreError: 164191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339467 with bad export cookie 5286797232006978692 [ 5341.451770] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5341.457348] LustreError: 164191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5341.699662] Lustre: server umount lustre-MDT0001 complete [ 5354.289575] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 5357.044764] Key type lgssc unregistered [ 5357.304679] LNet: 167391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5357.313394] LNetError: 167391:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5357.335417] LNet: Removed LNI 192.168.201.129@tcp [ 5358.059527] Key type .llcrypt unregistered [ 5358.063272] Key type ._llcrypt unregistered [ 5383.706037] Key type ._llcrypt registered [ 5383.708293] Key type .llcrypt registered [ 5383.844327] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 5385.073597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5385.170943] alg: No test for adler32 (adler32-zlib) [ 5386.310353] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 5386.546445] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 5388.239740] Key type lgssc registered [ 5389.439144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5402.148546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5402.168364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.564441] Lustre: *** cfs_fail_loc=135, val=0*** [ 5403.614293] LustreError: 168899:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5403.621838] LustreError: 168899:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.129@tcp: cfg command failed: rc = -2 [ 5403.631809] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5403.631809] [ 5403.638659] LustreError: MGC192.168.201.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5403.655581] LustreError: 168859:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5403.664295] LustreError: 168859:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -2 [ 5403.687061] LustreError: 168859:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5403.694243] LustreError: 168859:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5403.803594] Lustre: server umount lustre-MDT0000 complete [ 5403.807897] LustreError: 168859:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5410.239499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.521971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.143542] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5415.917380] LustreError: 169270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5415.942854] LustreError: 169270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5420.003788] LustreError: 169271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5423.011243] LustreError: 169270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5424.541494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5424.990358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5430.143570] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5435.711389] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5439.623613] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5445.601598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.615426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5445.615858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5445.616883] Lustre: Skipped 1 previous similar message [ 5445.646745] Lustre: Skipped 1 previous similar message [ 5449.697295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5454.816939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5454.820850] Lustre: Skipped 1 previous similar message [ 5456.351596] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5456.610417] Lustre: server umount lustre-MDT0000 complete [ 5459.937414] LustreError: 169982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.961980] LustreError: 169982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5460.947110] LustreError: 169251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339587 with bad export cookie 17796738298636785668 [ 5460.979814] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5461.546501] Lustre: server umount lustre-MDT0001 complete [ 5473.346889] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 5476.233776] Key type lgssc unregistered [ 5476.470353] LNet: 171349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5476.485177] LNetError: 171349:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5477.541421] LNet: Removed LNI 192.168.201.129@tcp [ 5478.312240] Key type .llcrypt unregistered [ 5478.314148] Key type ._llcrypt unregistered [ 5499.927634] Key type ._llcrypt registered [ 5499.929911] Key type .llcrypt registered [ 5500.052384] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 5500.506345] Key type .llcrypt unregistered [ 5500.511112] Key type ._llcrypt unregistered [ 5517.425316] Key type ._llcrypt registered [ 5517.428128] Key type .llcrypt registered [ 5517.540675] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 5518.545031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5518.815834] alg: No test for adler32 (adler32-zlib) [ 5519.877122] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 5520.038158] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 5521.671177] Key type lgssc registered [ 5522.625163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5530.463094] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5530.475922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5545.952389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5551.071624] LustreError: 173445:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 5577.698056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.770422] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5591.126221] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5598.270653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.449297] LustreError: 172782:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d3f1d415 x1866821424646400/t0(0) o250->MGC192.168.201.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5599.007694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.079683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5603.439723] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5604.328277] LustreError: 173470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5604.353765] LustreError: 173470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5609.445048] LustreError: 173471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5611.155602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.546059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5615.723482] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5621.991903] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5625.660846] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5628.606539] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5640.161632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5640.173921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5640.186067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5642.208092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5642.217591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.234691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5643.241614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5643.246678] Lustre: Skipped 1 previous similar message [ 5645.696662] Lustre: server umount lustre-OST0000 complete [ 5652.453759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5652.464325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.474586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5655.391675] Lustre: server umount lustre-MDT0000 complete [ 5658.593936] LustreError: 174373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5658.614432] LustreError: 174373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5659.430518] LustreError: 175772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339785 with bad export cookie 7196960477204132708 [ 5659.442767] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5659.443417] LustreError: 175772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5659.891251] Lustre: server umount lustre-MDT0001 complete [ 5671.863673] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 5674.060851] Key type lgssc unregistered [ 5674.352173] LNet: 177057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5674.360119] LNetError: 177057:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5674.378688] LNet: Removed LNI 192.168.201.129@tcp [ 5675.076950] Key type .llcrypt unregistered [ 5675.085427] Key type ._llcrypt unregistered [ 5697.922212] Key type ._llcrypt registered [ 5697.924788] Key type .llcrypt registered [ 5698.087901] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 5699.710876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5699.788516] alg: No test for adler32 (adler32-zlib) [ 5700.882761] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 5701.132666] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 5702.783237] Key type lgssc registered [ 5703.859782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5711.714370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5711.764592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5713.419606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5717.237101] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5722.599181] LustreError: 178452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5722.624615] LustreError: 178452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5724.889406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5725.313131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5729.694472] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5735.656727] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5739.850455] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5747.257889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5747.754668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5754.815423] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5758.014437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5762.927181] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5769.055492] Lustre: Failing over lustre-MDT0000 [ 5769.508459] Lustre: server umount lustre-MDT0000 complete [ 5770.951083] LustreError: 178446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.252423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5771.283408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.286686] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.287692] LustreError: 178451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.339622] Lustre: Skipped 1 previous similar message [ 5773.385163] LustreError: 178451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5775.993030] LustreError: 178446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5781.121552] LustreError: 179473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5781.138986] LustreError: 179473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5788.645449] Lustre: 177762:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780339899/real 1780339899] req@00000000e2e4bb09 x1866821614463360/t0(0) o400->MGC192.168.201.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780339915 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5788.682556] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5789.654827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5791.343900] LustreError: 178446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5791.381227] LustreError: 178446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5798.883407] LustreError: 177760:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d2d81466 x1866821614468736/t0(0) o250->MGC192.168.201.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5799.184709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5799.248868] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5801.575302] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5803.386926] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5804.519848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5804.532318] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5804.578993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5808.140754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5820.419425] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5830.630527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5830.640141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5830.665899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5831.648727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5831.665885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5833.008169] Lustre: server umount lustre-OST0000 complete [ 5840.354058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5840.365000] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5840.365574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5840.376629] Lustre: Skipped 2 previous similar messages [ 5845.474412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5845.478429] Lustre: Skipped 2 previous similar messages [ 5849.572395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5849.585891] Lustre: Skipped 1 previous similar message [ 5853.151208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5853.525201] Lustre: server umount lustre-MDT0000 complete [ 5854.690757] LustreError: 181737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5854.728414] LustreError: 181737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5858.367483] LustreError: 178432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339984 with bad export cookie 7335248466421816898 [ 5858.374952] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5858.383984] LustreError: 178432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5858.760376] Lustre: server umount lustre-MDT0001 complete [ 5870.944839] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 5874.112686] Key type lgssc unregistered [ 5874.445784] LNet: 183069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5874.450069] LNetError: 183069:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5874.477210] LNet: Removed LNI 192.168.201.129@tcp [ 5875.488647] Key type .llcrypt unregistered [ 5875.490490] Key type ._llcrypt unregistered [ 5902.112935] Key type ._llcrypt registered [ 5902.114722] Key type .llcrypt registered [ 5902.234780] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 5903.368828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5903.407273] alg: No test for adler32 (adler32-zlib) [ 5904.432759] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 5904.677188] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 5906.447767] Key type lgssc registered [ 5908.072446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5916.779636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5916.840717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5918.418085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5922.892940] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5927.923331] LustreError: 184465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5927.936390] LustreError: 184465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5931.630507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5932.269795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5937.233860] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5942.839862] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5947.176097] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5953.561616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5953.796629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5959.151125] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5963.077464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5966.020872] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5969.949269] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5974.265574] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5983.727562] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5993.952348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5993.959652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5993.966155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5993.988154] Lustre: Skipped 1 previous similar message [ 5999.076561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5999.094624] Lustre: Skipped 1 previous similar message [ 5999.551363] Lustre: server umount lustre-OST0000 complete [ 6003.682698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6003.705960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.733402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6003.736117] Lustre: Skipped 1 previous similar message [ 6009.313099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6009.323442] Lustre: Skipped 3 previous similar messages [ 6009.952817] Lustre: server umount lustre-MDT0000 complete [ 6014.433169] LustreError: 184460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.450160] LustreError: 184460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6014.725885] LustreError: 186661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340141 with bad export cookie 8577998737738953610 [ 6014.737942] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6014.750866] LustreError: 186661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6015.062718] Lustre: server umount lustre-MDT0001 complete [ 6025.559844] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6028.952835] Key type lgssc unregistered [ 6029.290889] LNet: 187812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6029.297106] LNetError: 187812:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6029.316179] LNet: Removed LNI 192.168.201.129@tcp [ 6030.328181] Key type .llcrypt unregistered [ 6030.330624] Key type ._llcrypt unregistered [ 6058.566535] Key type ._llcrypt registered [ 6058.569740] Key type .llcrypt registered [ 6058.694753] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 6061.090519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6061.175494] alg: No test for adler32 (adler32-zlib) [ 6062.255679] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 6062.475863] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 6064.183186] Key type lgssc registered [ 6065.769380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6074.861845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6074.918508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6076.670112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6081.657483] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6087.143885] LustreError: 189213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6087.176366] LustreError: 189213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6092.258464] LustreError: 189214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6092.287366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.728138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6098.608576] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6106.645215] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6112.593890] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6119.757833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6120.276888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6122.353057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6126.538595] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6132.564508] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6139.988815] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6154.694112] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6161.015591] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6166.870200] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6178.785820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6178.790736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.798426] LustreError: Skipped 1 previous similar message [ 6178.799652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6178.822073] Lustre: Skipped 1 previous similar message [ 6182.823733] Lustre: server umount lustre-OST0000 complete [ 6189.027111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6189.030932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6189.050456] Lustre: Skipped 1 previous similar message [ 6189.051599] Lustre: Skipped 1 previous similar message [ 6194.144971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6194.153801] Lustre: Skipped 2 previous similar messages [ 6199.268325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6199.280812] Lustre: Skipped 1 previous similar message [ 6201.311865] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6201.666768] Lustre: server umount lustre-MDT0000 complete [ 6204.387939] LustreError: 189213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.410242] LustreError: 189213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6206.736559] LustreError: 189932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340333 with bad export cookie 37647599966964200 [ 6206.747266] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6207.196974] Lustre: server umount lustre-MDT0001 complete [ 6220.121852] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6223.228741] Key type lgssc unregistered [ 6223.615331] LNet: 193737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6223.625478] LNetError: 193737:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6223.654309] LNet: Removed LNI 192.168.201.129@tcp [ 6224.776517] Key type .llcrypt unregistered [ 6224.779366] Key type ._llcrypt unregistered [ 6253.954894] Key type ._llcrypt registered [ 6253.958823] Key type .llcrypt registered [ 6254.181361] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 6255.453437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6255.562205] alg: No test for adler32 (adler32-zlib) [ 6256.794958] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 6256.993052] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 6258.663365] Key type lgssc registered [ 6260.194180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6268.722094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6268.747647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6270.570662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6275.694937] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6279.649081] LustreError: 195132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.683772] LustreError: 195132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6284.541466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6284.775900] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6284.936515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6289.457562] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6295.698350] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6299.696452] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6307.883950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6308.317650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6315.824652] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6316.558040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6321.308597] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6342.114667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6342.120449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6342.124885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6342.161100] Lustre: Skipped 1 previous similar message [ 6345.777982] Lustre: server umount lustre-OST0000 complete [ 6351.333342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6351.345094] LustreError: Skipped 1 previous similar message [ 6351.360269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6351.375679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6351.378046] Lustre: Skipped 1 previous similar message [ 6352.365094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6356.262182] Lustre: server umount lustre-MDT0000 complete [ 6357.473423] LustreError: 195129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6357.496380] LustreError: 195129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6360.059889] LustreError: 195114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340486 with bad export cookie 13998386164642773080 [ 6360.079062] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6360.079617] LustreError: 195114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6360.353508] Lustre: server umount lustre-MDT0001 complete [ 6373.951196] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6377.621619] Key type lgssc unregistered [ 6377.979371] LNet: 198476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6377.990431] LNetError: 198476:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6378.009111] LNet: Removed LNI 192.168.201.129@tcp [ 6379.193368] Key type .llcrypt unregistered [ 6379.197601] Key type ._llcrypt unregistered [ 6402.919620] Key type ._llcrypt registered [ 6402.921393] Key type .llcrypt registered [ 6403.041573] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 6404.319631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6404.340118] alg: No test for adler32 (adler32-zlib) [ 6405.523829] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 6405.800894] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 6407.607199] Key type lgssc registered [ 6409.203791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6421.233688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6421.266810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6422.986744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6429.620722] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6433.264263] LustreError: 199735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.306970] LustreError: 199735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6438.370810] LustreError: 199735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.349472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6440.053626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6445.595861] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6452.450814] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6456.818604] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6463.256273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6463.653964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6467.794265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6469.797214] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6475.565576] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6498.794576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6498.799841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6498.820013] Lustre: Skipped 1 previous similar message [ 6498.821525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6498.855669] Lustre: Skipped 1 previous similar message [ 6503.499946] Lustre: server umount lustre-OST0000 complete [ 6509.024523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.028640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6509.042422] Lustre: Skipped 1 previous similar message [ 6509.057754] Lustre: Skipped 1 previous similar message [ 6513.909457] Lustre: server umount lustre-MDT0000 complete [ 6514.154764] LustreError: 199736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6514.177108] LustreError: 199736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6517.531612] LustreError: 199717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340644 with bad export cookie 12462592551874841421 [ 6517.541133] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6517.812418] Lustre: server umount lustre-MDT0001 complete [ 6527.707767] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6530.067870] Key type lgssc unregistered [ 6530.253757] LNet: 203063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6530.258328] LNetError: 203063:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6530.266908] LNet: Removed LNI 192.168.201.129@tcp [ 6530.867238] Key type .llcrypt unregistered [ 6530.869520] Key type ._llcrypt unregistered [ 6549.439500] Key type ._llcrypt registered [ 6549.442640] Key type .llcrypt registered [ 6549.531027] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 6550.319990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6550.446065] alg: No test for adler32 (adler32-zlib) [ 6551.676202] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 6551.972126] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 6553.615951] Key type lgssc registered [ 6554.835567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6562.939222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6562.955968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6564.496100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6569.673295] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6574.053438] LustreError: 204304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6574.087864] LustreError: 204304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6579.113757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6579.183476] LustreError: 204303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.578267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6583.847110] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6590.605267] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6595.497756] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6603.224323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6603.836363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6608.945889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6610.795480] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6616.943318] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6627.031239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6627.078705] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6627.277070] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6634.108357] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6641.434679] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6643.602522] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6656.240347] Lustre: Permanently deactivating lustre-OST0001 [ 6656.273519] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6665.786464] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6671.483995] Lustre: Permanently reactivating lustre-OST0001 [ 6671.516713] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6671.519275] Lustre: Skipped 2 previous similar messages [ 6673.880477] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6673.902235] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6673.917723] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6673.934534] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6673.999252] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6683.435502] Lustre: server umount lustre-OST0001 complete [ 6683.627173] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6683.633144] Lustre: Skipped 2 previous similar messages [ 6683.636320] LustreError: 205901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.650634] LustreError: 205901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6688.738163] LustreError: 205900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6688.742437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6688.767506] LustreError: 205900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6688.778605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6693.861521] LustreError: 205899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6693.873331] LustreError: 205899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6693.953739] Lustre: server umount lustre-OST0000 complete [ 6697.453366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6697.465252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6697.479188] Lustre: Skipped 1 previous similar message [ 6697.490269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6697.494195] Lustre: Skipped 1 previous similar message [ 6698.977574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6698.989525] Lustre: Skipped 1 previous similar message [ 6703.913344] Lustre: server umount lustre-MDT0000 complete [ 6704.125445] LustreError: 204303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.163068] LustreError: 204303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6709.011514] LustreError: 204285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340835 with bad export cookie 9762683650662733121 [ 6709.039744] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6709.784516] Lustre: server umount lustre-MDT0001 complete [ 6723.080387] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6726.688185] Key type lgssc unregistered [ 6727.082430] LNet: 208884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6727.092582] LNetError: 208884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6727.123661] LNet: Removed LNI 192.168.201.129@tcp [ 6728.062442] Key type .llcrypt unregistered [ 6728.066926] Key type ._llcrypt unregistered [ 6772.138897] Key type ._llcrypt registered [ 6772.141310] Key type .llcrypt registered [ 6772.254867] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 6773.971869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6773.987791] alg: No test for adler32 (adler32-zlib) [ 6775.274462] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 6775.513548] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 6777.215263] Key type lgssc registered [ 6778.679110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6786.636415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6786.653579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6787.901833] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6787.949226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6788.147524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6792.662784] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6803.216428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.359053] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6803.442811] Lustre: 211076:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6803.898254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6808.799366] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6815.006923] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6819.742882] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6827.472605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6827.668934] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6827.917915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6834.323750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6834.800710] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6841.988207] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6845.110653] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6849.620649] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6856.728572] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6865.748334] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6879.096074] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6879.104865] Lustre: Skipped 1 previous similar message [ 6902.673931] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6902.687367] Lustre: Skipped 2 previous similar messages [ 6941.390442] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6941.399505] Lustre: Skipped 4 previous similar messages [ 7005.534530] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7005.540540] Lustre: Skipped 7 previous similar messages [ 7018.465246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7018.476518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7018.488625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7018.981521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7018.991948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7019.005129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7020.885825] Lustre: server umount lustre-OST0000 complete [ 7029.219846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7029.232371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7029.238800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7032.019856] Lustre: server umount lustre-MDT0000 complete [ 7036.671967] LustreError: 211068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341163 with bad export cookie 12677958004247365264 [ 7036.692488] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7037.339866] Lustre: server umount lustre-MDT0001 complete [ 7052.174361] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 7055.421980] Key type lgssc unregistered [ 7055.761647] LNet: 215516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7055.765692] LNetError: 215516:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7055.788861] LNet: Removed LNI 192.168.201.129@tcp [ 7056.502272] Key type .llcrypt unregistered [ 7056.504706] Key type ._llcrypt unregistered [ 7083.375554] Key type ._llcrypt registered [ 7083.382133] Key type .llcrypt registered [ 7083.584086] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 7085.175252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7085.212525] alg: No test for adler32 (adler32-zlib) [ 7087.006474] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 7087.487916] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 7089.263283] Key type lgssc registered [ 7090.539693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7098.115203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7098.139333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7099.636784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7105.064911] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7110.137787] LustreError: 216912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7110.173816] LustreError: 216912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7114.496794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7115.119178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7120.112834] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7126.150941] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7129.992824] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7137.543816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7138.165906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7145.276822] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7145.347244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7151.271501] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7169.456729] Lustre: server umount lustre-OST0000 complete [ 7176.167325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.183607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7176.191108] Lustre: Skipped 1 previous similar message [ 7176.215208] Lustre: Skipped 1 previous similar message [ 7181.280492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7181.285278] Lustre: Skipped 1 previous similar message [ 7183.495508] Lustre: server umount lustre-MDT0000 complete [ 7186.402447] LustreError: 217879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7186.439123] LustreError: 217879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7187.359749] Lustre: 216226:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780341297/real 1780341297] req@000000001af5fcb2 x1866823067798912/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780341313 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7187.394967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7187.424018] LustreError: 216908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7187.444143] LustreError: 216908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7187.876642] LustreError: 218531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341314 with bad export cookie 5675787570704018429 [ 7187.880222] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7187.895489] LustreError: 218531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7188.238746] Lustre: server umount lustre-MDT0001 complete [ 7199.586970] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 7202.580672] Key type lgssc unregistered [ 7202.876694] LNet: 220248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7202.892783] LNetError: 220248:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7202.912290] LNet: Removed LNI 192.168.201.129@tcp [ 7203.652841] Key type .llcrypt unregistered [ 7203.654668] Key type ._llcrypt unregistered [ 7226.580143] Key type ._llcrypt registered [ 7226.585343] Key type .llcrypt registered [ 7226.669441] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 7227.263481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7227.316611] alg: No test for adler32 (adler32-zlib) [ 7228.498260] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 7228.763206] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 7230.415262] Key type lgssc registered [ 7231.614724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7504.154399] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 7506.588720] Key type lgssc unregistered [ 7506.835907] LNet: 222242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7506.846276] LNetError: 222242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7506.867327] LNet: Removed LNI 192.168.201.129@tcp [ 7507.611796] Key type .llcrypt unregistered [ 7507.613976] Key type ._llcrypt unregistered [ 7529.348293] Key type ._llcrypt registered [ 7529.350535] Key type .llcrypt registered [ 7529.476927] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 7544.306623] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 7545.292520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7545.303756] alg: No test for adler32 (adler32-zlib) [ 7546.377277] Lustre: Lustre: Build Version: 2.17.53_35_g98ac157 [ 7546.617447] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 7548.247171] Key type lgssc registered [ 7549.161992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7574.151807] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7277 sec ========== 15:21:38 (1780341698) [ 7575.466466] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:21:40 (1780341700) === [ 7578.156402] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:21:43 (1780341703) === [ 7612.214271] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 7614.063311] Key type lgssc unregistered [ 7614.303960] LNet: 227426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7614.311736] LNetError: 227426:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7614.322551] LNet: Removed LNI 192.168.201.129@tcp [ 7614.900637] Key type .llcrypt unregistered [ 7614.904126] Key type ._llcrypt unregistered