[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481372491 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003152] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008621] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011153] LSM: Security Framework initializing [ 0.013063] Yama: becoming mindful. [ 0.014044] SELinux: Initializing. [ 0.015077] *** VALIDATE selinux *** [ 0.023358] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028475] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029128] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032132] *** VALIDATE tmpfs *** [ 0.034357] *** VALIDATE proc *** [ 0.035218] *** VALIDATE cgroup *** [ 0.036007] *** VALIDATE cgroup2 *** [ 0.037221] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040025] Spectre V2 : User space: Vulnerable [ 0.041008] Speculative Store Bypass: Vulnerable [ 0.044669] debug: unmapping init [mem 0xffffffff8b459000-0xffffffff8b460fff] [ 0.046190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047803] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049014] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052018] ... max period: 00007fffffffffff [ 0.053018] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055414] rcu: Hierarchical SRCU implementation. [ 0.057495] smp: Bringing up secondary CPUs ... [ 0.058486] x86: Booting SMP configuration: [ 0.059018] .... node #0, CPUs: #1 #2 #3 [ 0.062378] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065017] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.134611] node 0 deferred pages initialised in 68ms [ 0.137433] devtmpfs: initialized [ 0.138236] x86/mm: Memory block size: 128MB [ 0.140891] gcov: version magic: 0x41383552 [ 0.141655] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145260] pinctrl core: initialized pinctrl subsystem [ 0.147114] [ 0.147404] ************************************************************* [ 0.149012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151012] ** ** [ 0.153011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155014] ** ** [ 0.158015] ** This means that this kernel is built to expose internal ** [ 0.160012] ** IOMMU data structures, which may compromise security on ** [ 0.163020] ** your system. ** [ 0.165016] ** ** [ 0.167013] ** If you see this message and you are not debugging the ** [ 0.169012] ** kernel, report this immediately to your vendor! ** [ 0.172022] ** ** [ 0.174007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175006] ************************************************************* [ 0.176555] NET: Registered protocol family 16 [ 0.177271] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.182056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.185067] cpuidle: using governor menu [ 0.187632] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.189487] PCI: Using configuration type 1 for base access [ 0.191147] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.201065] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.203144] cryptd: max_cpu_qlen set to 1000 [ 0.205243] ACPI: Added _OSI(Module Device) [ 0.207016] ACPI: Added _OSI(Processor Device) [ 0.208014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215208] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.222008] ACPI: Interpreter enabled [ 0.223054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.224012] ACPI: Using IOAPIC for interrupt routing [ 0.225099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.228083] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.236550] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.238044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243100] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.248439] acpiphp: Slot [2] registered [ 0.250126] acpiphp: Slot [5] registered [ 0.252198] acpiphp: Slot [6] registered [ 0.253147] acpiphp: Slot [7] registered [ 0.254091] acpiphp: Slot [8] registered [ 0.255091] acpiphp: Slot [9] registered [ 0.258144] acpiphp: Slot [10] registered [ 0.259092] acpiphp: Slot [3] registered [ 0.260084] acpiphp: Slot [4] registered [ 0.261065] acpiphp: Slot [11] registered [ 0.262060] acpiphp: Slot [12] registered [ 0.264152] acpiphp: Slot [13] registered [ 0.265119] acpiphp: Slot [14] registered [ 0.266149] acpiphp: Slot [15] registered [ 0.268083] acpiphp: Slot [16] registered [ 0.269000] acpiphp: Slot [17] registered [ 0.269000] acpiphp: Slot [18] registered [ 0.269000] acpiphp: Slot [19] registered [ 0.271076] acpiphp: Slot [20] registered [ 0.271981] acpiphp: Slot [21] registered [ 0.273059] acpiphp: Slot [22] registered [ 0.274026] acpiphp: Slot [23] registered [ 0.274948] acpiphp: Slot [24] registered [ 0.275087] acpiphp: Slot [25] registered [ 0.276007] acpiphp: Slot [26] registered [ 0.276922] acpiphp: Slot [27] registered [ 0.278065] acpiphp: Slot [28] registered [ 0.278884] acpiphp: Slot [29] registered [ 0.280075] acpiphp: Slot [30] registered [ 0.281012] acpiphp: Slot [31] registered [ 0.282023] PCI host bridge to bus 0000:00 [ 0.282933] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.283025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.285020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.287022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.288024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.290029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.291147] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.294038] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.297554] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.305019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.311485] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.314021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.316016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.318016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.320470] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.322539] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.323029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.325611] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.329012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.335943] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.339012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.343886] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.349016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.354013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.368019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.383258] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.389017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.394012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.408018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.415815] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.422016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.428017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.442019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.449640] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.458017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.462013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.473014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.481130] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.486013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.490012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.504081] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.512181] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.516017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.522015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.537018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.547814] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.549337] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.551298] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.553481] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.556217] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.560261] iommu: Default domain type: Passthrough [ 0.562349] SCSI subsystem initialized [ 0.563117] ACPI: bus type USB registered [ 0.564085] usbcore: registered new interface driver usbfs [ 0.565105] usbcore: registered new interface driver hub [ 0.567059] usbcore: registered new device driver usb [ 0.569158] pps_core: LinuxPPS API ver. 1 registered [ 0.570007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.573054] PTP clock support registered [ 0.574139] EDAC MC: Ver: 3.0.0 [ 0.576085] PCI: Using ACPI for IRQ routing [ 0.577572] NetLabel: Initializing [ 0.578010] NetLabel: domain hash size = 128 [ 0.578882] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.580071] NetLabel: unlabeled traffic allowed by default [ 0.582103] vgaarb: loaded [ 0.583257] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.585012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.590979] clocksource: Switched to clocksource kvm-clock [ 0.696557] VFS: Disk quotas dquot_6.6.0 [ 0.697811] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.699821] *** VALIDATE ramfs *** [ 0.700806] *** VALIDATE hugetlbfs *** [ 0.701974] pnp: PnP ACPI init [ 0.703881] pnp: PnP ACPI: found 6 devices [ 0.725051] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.727606] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.729305] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.730843] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.732739] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.734646] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.736960] NET: Registered protocol family 2 [ 0.739058] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.743114] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.745824] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.750219] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.753077] TCP: Hash tables configured (established 65536 bind 65536) [ 0.755443] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.757954] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.760060] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.762311] NET: Registered protocol family 1 [ 0.765833] RPC: Registered named UNIX socket transport module. [ 0.767620] RPC: Registered udp transport module. [ 0.768996] RPC: Registered tcp transport module. [ 0.770254] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.772028] NET: Registered protocol family 44 [ 0.773679] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.775990] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.778016] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.780399] PCI: CLS 0 bytes, default 64 [ 0.781974] Unpacking initramfs... [ 2.150404] debug: unmapping init [mem 0xffff8ee93cc54000-0xffff8ee93ffbffff] [ 2.153634] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.155185] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.157465] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.594329] Initialise system trusted keyrings [ 2.596170] Key type blacklist registered [ 2.599098] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.611233] zbud: loaded [ 2.614878] *** VALIDATE nfs *** [ 2.616462] *** VALIDATE nfs4 *** [ 2.618141] pstore: using deflate compression [ 2.621653] Platform Keyring initialized [ 2.702716] NET: Registered protocol family 38 [ 2.704372] Key type asymmetric registered [ 2.705684] Asymmetric key parser 'x509' registered [ 2.707405] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.711019] io scheduler mq-deadline registered [ 2.712574] io scheduler kyber registered [ 2.714431] io scheduler bfq registered [ 2.716331] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.719581] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.721997] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.724090] ACPI: Power Button [PWRF] [ 2.728639] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.734535] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.745933] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.753227] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.767977] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.794665] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.821994] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.825310] Non-volatile memory driver v1.3 [ 2.826419] Linux agpgart interface v0.103 [ 2.849462] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.851596] vda: detected capacity change from 0 to 76791808 [ 2.865338] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.867814] vdb: detected capacity change from 0 to 1073741824 [ 2.879747] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.882513] vdc: detected capacity change from 0 to 2621440000 [ 2.900249] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.902631] vdd: detected capacity change from 0 to 2621440000 [ 2.915544] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.917674] vde: detected capacity change from 0 to 4294967296 [ 2.930583] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.932824] vdf: detected capacity change from 0 to 4294967296 [ 2.943504] libphy: Fixed MDIO Bus: probed [ 2.952967] usbcore: registered new interface driver usbserial_generic [ 2.955685] usbserial: USB Serial support registered for generic [ 2.958475] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.963079] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.965302] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.967855] mousedev: PS/2 mouse device common for all mice [ 2.970292] rtc_cmos 00:05: RTC can wake from S4 [ 2.973662] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.974039] rtc_cmos 00:05: registered as rtc0 [ 2.979216] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.980721] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.983110] intel_pstate: CPU model not supported [ 2.989301] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.994423] hid: raw HID events driver (C) Jiri Kosina [ 2.996534] usbcore: registered new interface driver usbhid [ 2.998282] usbhid: USB HID core driver [ 2.999812] drop_monitor: Initializing network drop monitor service [ 3.003934] Initializing XFRM netlink socket [ 3.005563] NET: Registered protocol family 10 [ 3.007807] Segment Routing with IPv6 [ 3.009071] NET: Registered protocol family 17 [ 3.011134] mpls_gso: MPLS GSO support [ 3.015540] RAS: Correctable Errors collector initialized. [ 3.017139] AVX version of gcm_enc/dec engaged. [ 3.018309] AES CTR mode by8 optimization enabled [ 3.076281] sched_clock: Marking stable (3076252804, 0)->(3959583350, -883330546) [ 3.079093] registered taskstats version 1 [ 3.080370] Loading compiled-in X.509 certificates [ 3.081735] zswap: loaded using pool lzo/zbud [ 3.103884] Key type big_key registered [ 3.112118] Key type encrypted registered [ 3.113887] ima: No TPM chip found, activating TPM-bypass! [ 3.116271] ima: Allocated hash algorithm: sha1 [ 3.118170] ima: No architecture policies found [ 3.120215] evm: Initialising EVM extended attributes: [ 3.122225] evm: security.selinux [ 3.123571] evm: security.ima [ 3.124894] evm: security.capability [ 3.126100] evm: HMAC attrs: 0x1 [ 3.128540] rtc_cmos 00:05: setting system clock to 2026-09-09 16:09:05 UTC (1788970145) [ 3.134607] debug: unmapping init [mem 0xffffffff8c403000-0xffffffff8c5fffff] [ 3.138212] debug: unmapping init [mem 0xffffffff8b182000-0xffffffff8b458fff] [ 3.148094] Write protecting the kernel read-only data: 28672k [ 3.151967] debug: unmapping init [mem 0xffffffff89803000-0xffffffff899fffff] [ 3.154872] debug: unmapping init [mem 0xffffffff8a114000-0xffffffff8a1fffff] [ 3.190103] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.197906] systemd[1]: Detected virtualization kvm. [ 3.199835] systemd[1]: Detected architecture x86-64. [ 3.201972] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.224963] systemd[1]: No hostname configured. [ 3.226787] systemd[1]: Set hostname to . [ 3.228968] random: systemd: uninitialized urandom read (16 bytes read) [ 3.231591] systemd[1]: Initializing machine ID from random generator. [ 3.352342] random: systemd: uninitialized urandom read (16 bytes read) [ 3.354155] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.358556] random: systemd: uninitialized urandom read (16 bytes read) [ 3.361298] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.365597] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.921599] device-mapper: uevent: version 1.0.3 [ 3.923542] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.516714] random: fast init done [ 4.555738] virtio_net virtio0 ens2: renamed from eth0 [ 4.605470] scsi host0: ata_piix [ 4.622194] scsi host1: ata_piix [ 4.641360] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.643796] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.529306] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.666914] random: crng init done [ 9.668317] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.894350] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.608871] printk: systemd: 24 output lines suppressed due to ratelimiting [ 14.170185] SELinux: Disabled at runtime. [ 14.301811] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.325373] systemd[1]: Detected virtualization kvm. [ 14.327327] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.049169] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.055780] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.067982] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.074079] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.079941] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.095805] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.130938] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 16.220134] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started udev Coldplug all Devices. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 17.575054] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.625547] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.779081] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.208553] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.292333] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 25.426759] Key type dns_resolver registered [ ***] A start job is running for Configur…-only root support (9s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit)[ 26.116130] NFS: Registering the id_resolver key type [ 26.120903] Key type id_resolver registered [ 26.125696] Key type id_legacy registered [ *] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg115-server login: [ 60.167236] hrtimer: interrupt took 2449873 ns [ 100.427050] libcfs: loading out-of-tree module taints kernel. [ 100.487802] Key type ._llcrypt registered [ 100.490920] Key type .llcrypt registered [ 100.616797] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 125.394340] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 126.817776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 126.832764] alg: No test for adler32 (adler32-zlib) [ 128.209910] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 129.222767] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 131.143290] Key type lgssc registered [ 133.156685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.848454] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 199.316311] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 213.087693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 213.138049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.369694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 214.416797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 214.608774] Lustre: lustre-MDT0000: new disk, initializing [ 214.791196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.825490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 219.740979] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.185444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 234.305516] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 234.342322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 234.346456] Lustre: Skipped 1 previous similar message [ 234.435279] Lustre: lustre-MDT0001: new disk, initializing [ 234.579089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 234.612585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 234.623230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 239.263501] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.608563] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 254.969869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 255.413534] Lustre: lustre-OST0000: new disk, initializing [ 255.417266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 255.431994] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 255.512794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.621241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 259.633101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.722412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 261.646258] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.966956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 276.038856] Lustre: lustre-OST0001: new disk, initializing [ 276.044668] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 276.048747] Lustre: 9527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 276.109441] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 282.674519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 282.685830] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 282.751070] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 282.791628] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 293.994868] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 306.448867] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 315.905638] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing check_logdir /tmp/testlogs/ [ 321.640851] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing yml_node [ 327.134397] Lustre: DEBUG MARKER: Client: 2.17.58.42 [ 330.090617] Lustre: DEBUG MARKER: MDS: 2.17.58.42 [ 332.848410] Lustre: DEBUG MARKER: OSS: 2.17.58.42 [ 334.672031] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 9 12:14:36 EDT 2026 [ 351.296766] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 352.936061] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 369.648636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.656148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.664770] Lustre: Skipped 1 previous similar message [ 369.672544] Lustre: Skipped 2 previous similar messages [ 374.754869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 374.767560] Lustre: Skipped 4 previous similar messages [ 375.490565] Lustre: server umount lustre-MDT0000 complete [ 379.799502] LustreError: 8464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970522 with bad export cookie 12501329666361305313 [ 379.806548] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 379.824567] LustreError: 8464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 379.887584] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.888780] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.910892] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 379.912420] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 379.945890] Lustre: Skipped 3 previous similar messages [ 384.992641] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 384.992711] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 384.992716] Lustre: Skipped 1 previous similar message [ 385.031761] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 390.113305] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.114303] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 390.139291] Lustre: Skipped 2 previous similar messages [ 394.207171] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 394.463834] Lustre: server umount lustre-MDT0001 complete [ 398.680513] Lustre: server umount lustre-OST0000 complete [ 402.582681] Lustre: server umount lustre-OST0001 complete [ 408.953426] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 419.467648] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 457.390522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.635315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 457.665498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 457.799738] Lustre: lustre-MDT0000: new disk, initializing [ 457.989926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 458.009628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 463.410902] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 474.730486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.852797] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 474.927718] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 474.931829] Lustre: Skipped 1 previous similar message [ 475.038850] Lustre: lustre-MDT0001: new disk, initializing [ 475.153866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 475.195839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 475.210776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 479.907698] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 492.794663] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 498.692713] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 505.959880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 506.226501] Lustre: lustre-OST0000: new disk, initializing [ 506.229818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 506.238674] Lustre: 17046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 506.341534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.315343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 508.323608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 508.525740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 514.469600] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 524.736254] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 530.579059] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 530.816695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 536.080938] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 536.304556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 539.108758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.124581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.146995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.776376] Lustre: server umount lustre-OST0000 complete [ 549.345428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.373616] Lustre: Skipped 1 previous similar message [ 555.458789] Lustre: server umount lustre-MDT0000 complete [ 559.151574] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970701 with bad export cookie 12501329666361306853 [ 559.162270] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 559.540955] Lustre: server umount lustre-MDT0001 complete [ 564.014878] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 12:18:24 (1788970704) [ 571.391969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.840073] LustreError: 19311:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.867700] LustreError: 19311:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 571.931384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.995410] LustreError: 19312:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.167513] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 586.652820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 592.921875] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 601.531746] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 606.853939] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 614.944597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 620.004246] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 627.619378] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 638.947987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 638.955199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.963530] Lustre: Skipped 1 previous similar message [ 638.974237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 638.983237] Lustre: Skipped 7 previous similar messages [ 643.049900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 644.741903] Lustre: server umount lustre-OST0000 complete [ 648.677622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 648.687582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.695287] Lustre: Skipped 1 previous similar message [ 655.093559] Lustre: server umount lustre-MDT0000 complete [ 658.403113] LustreError: 19308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.429487] LustreError: 19308:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 659.854451] LustreError: 19293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970802 with bad export cookie 12501329666361307735 [ 659.885956] LustreError: 19293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 659.891582] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.574903] Lustre: server umount lustre-MDT0001 complete [ 676.177537] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 679.152837] Key type lgssc unregistered [ 679.554315] LNet: 22845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.575207] LNetError: 22845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.588728] LNet: Removed LNI 192.168.201.115@tcp [ 680.640171] Key type .llcrypt unregistered [ 680.642590] Key type ._llcrypt unregistered [ 709.409187] Key type ._llcrypt registered [ 709.412234] Key type .llcrypt registered [ 709.526621] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 710.921712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 710.949532] alg: No test for adler32 (adler32-zlib) [ 712.469663] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 712.782299] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 714.471450] Key type lgssc registered [ 715.779857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 723.945507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 723.952883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.548253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 730.708804] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 734.703132] LustreError: 24252:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 734.731991] LustreError: 24252:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 739.525535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 739.847426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 743.933227] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 751.346510] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 758.033608] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 764.890210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 765.398296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 770.557345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 772.328480] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 781.025787] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 796.128435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 796.133120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.154660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 796.199281] Lustre: Skipped 1 previous similar message [ 797.649465] Lustre: server umount lustre-OST0000 complete [ 806.375655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 806.397133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 806.403599] Lustre: Skipped 1 previous similar message [ 806.412511] Lustre: Skipped 2 previous similar messages [ 806.423939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 808.211185] Lustre: server umount lustre-MDT0000 complete [ 811.500488] LustreError: 26131:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 811.539891] LustreError: 26131:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 812.563322] LustreError: 24233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970954 with bad export cookie 2542669061517682153 [ 812.576598] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 812.849829] Lustre: server umount lustre-MDT0001 complete [ 824.911356] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 827.351438] Key type lgssc unregistered [ 827.681620] LNet: 28225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.694772] LNetError: 28225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.711072] LNet: Removed LNI 192.168.201.115@tcp [ 829.733379] Key type .llcrypt unregistered [ 829.735360] Key type ._llcrypt unregistered [ 855.910411] Key type ._llcrypt registered [ 855.915315] Key type .llcrypt registered [ 856.046214] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 857.517526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 857.589742] alg: No test for adler32 (adler32-zlib) [ 858.718795] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 859.056613] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 860.831527] Key type lgssc registered [ 862.255613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 871.242905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 871.261492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.663121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 878.279808] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 883.169865] LustreError: 29634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 883.187110] LustreError: 29634:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 887.623970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.057165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 892.788739] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 898.915824] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 905.241884] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 919.820271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 920.152679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.976744] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 935.208982] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 942.707406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 958.434775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.459206] Lustre: Skipped 1 previous similar message [ 958.466702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 958.484884] Lustre: Skipped 1 previous similar message [ 962.616249] Lustre: server umount lustre-OST0000 complete [ 968.674379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.678301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 968.693673] Lustre: Skipped 1 previous similar message [ 968.717210] Lustre: Skipped 1 previous similar message [ 973.804311] LustreError: 29635:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.821634] LustreError: 29635:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 973.843921] Lustre: server umount lustre-MDT0000 complete [ 977.890305] LustreError: 29630:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 977.905145] LustreError: 29630:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 978.207165] LustreError: 29615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971120 with bad export cookie 5999404478214690729 [ 978.214769] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 978.229058] LustreError: 29615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 978.533356] Lustre: server umount lustre-MDT0001 complete [ 990.914497] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 993.970353] Key type lgssc unregistered [ 994.334292] LNet: 33543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.351408] LNetError: 33543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.385428] LNet: Removed LNI 192.168.201.115@tcp [ 995.410533] Key type .llcrypt unregistered [ 995.416089] Key type ._llcrypt unregistered [ 1018.862608] Key type ._llcrypt registered [ 1018.867094] Key type .llcrypt registered [ 1019.030661] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1020.235726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1020.351602] alg: No test for adler32 (adler32-zlib) [ 1021.784719] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1022.124721] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1023.863302] Key type lgssc registered [ 1025.559514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1035.977144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1036.009181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.740733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.548353] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1048.042019] LustreError: 34960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.071288] LustreError: 34960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1053.156153] LustreError: 34961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.315738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1053.744406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1058.942921] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1068.104383] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1074.256579] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1082.745450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1083.284411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.303532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1090.825404] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1099.133097] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1109.985362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.989131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1110.018879] Lustre: Skipped 1 previous similar message [ 1110.035377] Lustre: Skipped 1 previous similar message [ 1114.472401] Lustre: server umount lustre-OST0000 complete [ 1120.228686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.236905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1120.260909] Lustre: Skipped 1 previous similar message [ 1120.263820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1120.270065] Lustre: Skipped 1 previous similar message [ 1125.345160] LustreError: 37451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.365939] LustreError: 37451:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1125.402874] Lustre: server umount lustre-MDT0000 complete [ 1129.328107] LustreError: 34941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971271 with bad export cookie 10257665582760685789 [ 1129.344946] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1129.875337] Lustre: server umount lustre-MDT0001 complete [ 1143.603736] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1147.237747] Key type lgssc unregistered [ 1147.563114] LNet: 38491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.583838] LNetError: 38491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1147.620579] LNet: Removed LNI 192.168.201.115@tcp [ 1148.637454] Key type .llcrypt unregistered [ 1148.639879] Key type ._llcrypt unregistered [ 1174.574585] Key type ._llcrypt registered [ 1174.584597] Key type .llcrypt registered [ 1174.738872] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1175.859880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1175.891985] alg: No test for adler32 (adler32-zlib) [ 1177.290713] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1177.528307] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1179.239274] Key type lgssc registered [ 1180.601966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1188.962483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1188.984358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.437310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.657948] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1200.620143] LustreError: 39904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.651314] LustreError: 39904:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1205.283050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.677784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1211.202655] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1219.990429] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1226.583311] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1234.423309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1234.996716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.018757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1243.671425] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1252.249388] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1260.070210] Lustre: lustre-OST0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 1260.512690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1260.535308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.556541] Lustre: Skipped 1 previous similar message [ 1265.161678] Lustre: lustre-OST0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 1265.178152] Lustre: Skipped 2 previous similar messages [ 1265.522471] Lustre: server umount lustre-OST0000 complete [ 1286.117293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.122945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1286.138466] Lustre: Skipped 1 previous similar message [ 1286.161876] Lustre: Skipped 1 previous similar message [ 1291.237082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1291.250031] Lustre: Skipped 1 previous similar message [ 1292.041875] Lustre: server umount lustre-MDT0000 complete [ 1296.283581] LustreError: 39884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971438 with bad export cookie 15518422823673238449 [ 1296.289073] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1296.299069] LustreError: 39884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1302.748705] Lustre: server umount lustre-MDT0001 complete [ 1315.627673] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1318.828139] Key type lgssc unregistered [ 1319.158047] LNet: 43614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1319.166338] LNetError: 43614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.182795] LNet: Removed LNI 192.168.201.115@tcp [ 1320.192461] Key type .llcrypt unregistered [ 1320.194953] Key type ._llcrypt unregistered [ 1345.006912] Key type ._llcrypt registered [ 1345.009770] Key type .llcrypt registered [ 1345.196635] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1346.590580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1346.721642] alg: No test for adler32 (adler32-zlib) [ 1347.800379] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1348.084126] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1349.831232] Key type lgssc registered [ 1351.225225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1360.577302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1360.610514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.318722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1367.715176] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1372.649903] LustreError: 45024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.675858] LustreError: 45024:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1376.031172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.308940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1381.106593] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1390.692143] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1397.586776] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1405.828697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1406.242646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.315322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1413.192080] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1421.978366] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1429.822070] Lustre: server umount lustre-MDT0000 complete [ 1430.067950] LustreError: 46016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1432.045309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.060882] LustreError: 47425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1432.073766] Lustre: Skipped 2 previous similar messages [ 1432.096036] LustreError: 47425:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1433.966194] LustreError: 45005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971576 with bad export cookie 6873399839419685937 [ 1433.998738] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1435.158759] LustreError: 45019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.192619] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [ 1437.157417] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.186737] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1440.268797] LustreError: 46016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1440.286154] LustreError: 46016:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1440.449405] Lustre: server umount lustre-MDT0001 complete [ 1461.974023] Lustre: server umount lustre-OST0000 complete [ 1478.338234] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1481.635531] Key type lgssc unregistered [ 1481.879688] LNet: 48882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.882663] LNetError: 48882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.894518] LNet: Removed LNI 192.168.201.115@tcp [ 1482.738078] Key type .llcrypt unregistered [ 1482.740103] Key type ._llcrypt unregistered [ 1514.142498] Key type ._llcrypt registered [ 1514.144735] Key type .llcrypt registered [ 1514.245526] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1515.759350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1515.775237] alg: No test for adler32 (adler32-zlib) [ 1516.832615] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1517.060142] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1518.721197] Key type lgssc registered [ 1519.616716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1527.698819] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1527.717755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1543.266783] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1548.383322] LustreError: 50269:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [ 1575.199360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.893831] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1587.799827] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1755.052518] Lustre: server umount lustre-OST0000 complete [ 1772.374657] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1775.531852] Key type lgssc unregistered [ 1775.932928] LNet: 51942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.939134] LNetError: 51942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.954922] LNet: Removed LNI 192.168.201.115@tcp [ 1776.798728] Key type .llcrypt unregistered [ 1776.799973] Key type ._llcrypt unregistered [ 1800.529335] Key type ._llcrypt registered [ 1800.530725] Key type .llcrypt registered [ 1800.621593] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1802.059348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1802.109710] alg: No test for adler32 (adler32-zlib) [ 1803.198373] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1803.546176] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1805.263708] Key type lgssc registered [ 1806.508845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1815.021858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1815.051784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.706413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.542906] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1827.305698] LustreError: 53354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1827.320331] LustreError: 53354:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1830.627503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.073097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1835.448531] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1842.658740] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1847.100924] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1854.817362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1855.354800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.595048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1863.119522] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1873.390988] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1877.984807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.990959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.991405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1878.012932] Lustre: Skipped 1 previous similar message [ 1878.041038] Lustre: Skipped 1 previous similar message [ 1882.662422] Lustre: server umount lustre-OST0000 complete [ 1887.201384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1887.215458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1887.237197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.229726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1888.245239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.248109] Lustre: Skipped 1 previous similar message [ 1892.908805] Lustre: server umount lustre-MDT0000 complete [ 1893.344730] LustreError: 53355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1893.366278] LustreError: 53355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1897.444397] LustreError: 53334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972039 with bad export cookie 14401972449191427251 [ 1897.468962] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1897.972492] Lustre: server umount lustre-MDT0001 complete [ 1913.056641] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1916.768742] Key type lgssc unregistered [ 1917.089681] LNet: 56886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.098276] LNetError: 56886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1918.124308] LNet: Removed LNI 192.168.201.115@tcp [ 1919.209460] Key type .llcrypt unregistered [ 1919.212221] Key type ._llcrypt unregistered [ 1940.531453] Key type ._llcrypt registered [ 1940.535845] Key type .llcrypt registered [ 1940.687371] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1942.014131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1942.042454] alg: No test for adler32 (adler32-zlib) [ 1943.196531] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1943.472441] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1945.183315] Key type lgssc registered [ 1946.328222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1953.976833] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1954.011286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1969.567354] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1974.687391] LustreError: 58245:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [ 2001.439912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.420336] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2016.670726] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2023.543105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.648868] LustreError: 57581:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e3d21f88 x1875873193003264/t0(0) o250->MGC192.168.201.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2024.053643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2024.115927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2028.694556] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2029.029257] LustreError: 58270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.052145] LustreError: 58270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2034.150619] LustreError: 58271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.193707] LustreError: 58270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.200917] LustreError: 58270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2037.222305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.537130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2042.991388] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2051.773258] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2057.577469] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2059.744691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2059.749344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.755956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2063.330246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2063.348726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2063.354376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.399328] Lustre: Skipped 1 previous similar message [ 2065.597625] Lustre: server umount lustre-OST0000 complete [ 2087.907163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.927632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2087.932493] Lustre: Skipped 1 previous similar message [ 2087.945981] Lustre: Skipped 1 previous similar message [ 2092.278252] Lustre: server umount lustre-MDT0000 complete [ 2093.025932] LustreError: 59286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.044129] LustreError: 59286:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2095.755734] LustreError: 58253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972238 with bad export cookie 9919062920270169395 [ 2095.767854] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2102.359828] Lustre: server umount lustre-MDT0001 complete [ 2113.026751] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2115.523383] Key type lgssc unregistered [ 2115.770687] LNet: 61988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.790442] LNetError: 61988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.839473] LNet: Removed LNI 192.168.201.115@tcp [ 2117.510093] Key type .llcrypt unregistered [ 2117.512559] Key type ._llcrypt unregistered [ 2138.372239] Key type ._llcrypt registered [ 2138.374305] Key type .llcrypt registered [ 2138.469360] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2139.821736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2139.831656] alg: No test for adler32 (adler32-zlib) [ 2140.882637] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2141.083666] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2142.776087] Key type lgssc registered [ 2143.832175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2153.972533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2154.010700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2155.798282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.526203] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2166.252349] LustreError: 63398:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.292025] LustreError: 63398:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2171.369973] LustreError: 63399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.739386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.069254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2177.523756] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2184.809551] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2189.935328] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2195.730914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2196.238249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.371620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2203.046765] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2212.896861] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2232.288636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2232.295752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2232.307028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2232.344460] Lustre: Skipped 1 previous similar message [ 2237.414674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2237.419061] Lustre: Skipped 2 previous similar messages [ 2242.529040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2242.548233] Lustre: Skipped 1 previous similar message [ 2246.113782] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2246.355576] Lustre: server umount lustre-OST0000 complete [ 2252.770858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.772731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.783318] Lustre: Skipped 1 previous similar message [ 2256.087045] Lustre: server umount lustre-MDT0000 complete [ 2257.888638] LustreError: 63395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.895729] LustreError: 63395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2259.394381] LustreError: 63378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972401 with bad export cookie 10242403372284004123 [ 2259.406081] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2259.408508] LustreError: 63378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2259.835289] Lustre: server umount lustre-MDT0001 complete [ 2271.111614] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2274.534373] Key type lgssc unregistered [ 2274.913635] LNet: 66954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.927780] LNetError: 66954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.940216] LNet: Removed LNI 192.168.201.115@tcp [ 2275.945205] Key type .llcrypt unregistered [ 2275.947918] Key type ._llcrypt unregistered [ 2301.747845] Key type ._llcrypt registered [ 2301.750718] Key type .llcrypt registered [ 2301.863657] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2302.913803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2302.925694] alg: No test for adler32 (adler32-zlib) [ 2304.026851] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2304.315121] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2305.983765] Key type lgssc registered [ 2307.348809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2317.131592] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 12:47:38 (1788972458) [ 2325.375893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2325.443336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.373565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.651978] LustreError: 68656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.696345] LustreError: 68656:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2332.785947] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2336.745533] LustreError: 68657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.861958] LustreError: 68656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.225328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2343.680478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2348.716134] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2356.760667] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2364.395689] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2371.552717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2371.926260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2373.014458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2378.714646] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2386.656672] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2393.725742] Lustre: Failing over lustre-MDT0000 [ 2394.588880] Lustre: server umount lustre-MDT0000 complete [ 2394.719150] LustreError: 70091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.109329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2395.122987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.694615] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.719720] LustreError: 70091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.736429] LustreError: 70091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2406.161634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.278133] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2406.409539] Lustre: *** cfs_fail_loc=135, val=0*** [ 2406.566436] LustreError: 71746:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2406.581250] LustreError: 71746:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.115@tcp: cfg command failed: rc = -2 [ 2406.592096] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2406.592096] [ 2406.598525] LustreError: MGC192.168.201.115@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2406.611265] LustreError: 71734:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2406.628489] LustreError: 71734:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2406.634762] LustreError: 71734:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2406.647414] LustreError: 71734:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2406.664795] Lustre: server umount lustre-MDT0000 complete [ 2406.671267] LustreError: 71734:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2408.932044] LustreError: 68656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.945594] LustreError: 68656:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2414.429314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.560868] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2414.884160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.949894] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2415.186271] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2420.080956] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2420.215762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2420.267688] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2420.340721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2429.411527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2429.425210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.445405] Lustre: Skipped 1 previous similar message [ 2429.454602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.439063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.447042] Lustre: Skipped 1 previous similar message [ 2435.558384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2435.575584] Lustre: Skipped 1 previous similar message [ 2440.677253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2440.682940] Lustre: Skipped 1 previous similar message [ 2442.207195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2442.407915] Lustre: server umount lustre-OST0000 complete [ 2450.917040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.918926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2450.924874] Lustre: Skipped 1 previous similar message [ 2450.925965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.938931] Lustre: Skipped 1 previous similar message [ 2452.442833] Lustre: server umount lustre-MDT0000 complete [ 2455.813169] LustreError: 68637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972598 with bad export cookie 11138673166262899201 [ 2455.817915] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2455.829620] LustreError: 68637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2456.304522] Lustre: server umount lustre-MDT0001 complete [ 2470.725730] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2474.135244] Key type lgssc unregistered [ 2474.422923] LNet: 73524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.437103] LNetError: 73524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.467479] LNet: Removed LNI 192.168.201.115@tcp [ 2475.683827] Key type .llcrypt unregistered [ 2475.693122] Key type ._llcrypt unregistered [ 2501.152995] Key type ._llcrypt registered [ 2501.158376] Key type .llcrypt registered [ 2501.300266] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2502.733734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2502.771320] alg: No test for adler32 (adler32-zlib) [ 2503.851872] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2504.265817] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2506.023309] Key type lgssc registered [ 2507.117624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2514.604901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2514.619453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.310894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.578867] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2525.666429] LustreError: 74936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.700832] LustreError: 74936:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2528.422830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.121608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2536.320339] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2545.770754] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2552.216245] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2559.130823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2559.462562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.545368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2565.652399] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2574.756621] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2582.297862] Lustre: Failing over lustre-MDT0000 [ 2582.575563] LustreError: 74932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.832783] Lustre: server umount lustre-MDT0000 complete [ 2584.036780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.045527] LustreError: 74936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.048708] Lustre: Skipped 1 previous similar message [ 2584.120657] LustreError: 74936:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2587.664530] LustreError: 76032:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.771557] LustreError: 74932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.788056] LustreError: 74932:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2593.312440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.572799] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2593.787302] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2593.900594] LustreError: 78023:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2593.908765] LustreError: 78023:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.115@tcp: cfg command failed: rc = -9 [ 2593.926342] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2593.926342] [ 2593.935458] LustreError: MGC192.168.201.115@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2593.949855] LustreError: 78012:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2593.959207] LustreError: 78012:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2593.968238] LustreError: 78012:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2593.974667] LustreError: 78012:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2594.017451] Lustre: server umount lustre-MDT0000 complete [ 2594.021746] LustreError: 78012:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2602.706408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.981385] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.005459] LustreError: 78370:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.115@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000336cfdad) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2603.049772] LustreError: 74929:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2603.611303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2603.741529] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2604.039336] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2608.636350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2608.668340] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2608.759717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2608.833486] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2618.849951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.865284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2618.870515] Lustre: Skipped 2 previous similar messages [ 2618.890126] Lustre: Skipped 1 previous similar message [ 2623.939692] Lustre: server umount lustre-OST0000 complete [ 2629.088920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2629.092493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.113398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.130074] Lustre: Skipped 1 previous similar message [ 2629.149020] Lustre: Skipped 1 previous similar message [ 2634.212166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2634.225456] Lustre: Skipped 1 previous similar message [ 2635.377197] Lustre: server umount lustre-MDT0000 complete [ 2639.250086] LustreError: 74919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972781 with bad export cookie 17167350630743443362 [ 2639.253917] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2639.261611] LustreError: 74919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2639.704364] Lustre: server umount lustre-MDT0001 complete [ 2652.328946] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2655.962171] Key type lgssc unregistered [ 2656.439026] LNet: 79804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2656.449668] LNetError: 79804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2656.482913] LNet: Removed LNI 192.168.201.115@tcp [ 2658.101629] Key type .llcrypt unregistered [ 2658.104610] Key type ._llcrypt unregistered [ 2674.262832] Key type ._llcrypt registered [ 2674.266831] Key type .llcrypt registered [ 2674.378617] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2674.840636] Key type .llcrypt unregistered [ 2674.844137] Key type ._llcrypt unregistered [ 2701.554233] Key type ._llcrypt registered [ 2701.557321] Key type .llcrypt registered [ 2701.702847] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2703.199926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2703.342916] alg: No test for adler32 (adler32-zlib) [ 2704.397376] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2704.605685] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2706.295249] Key type lgssc registered [ 2707.856683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2714.937284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2714.978348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.848083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.935307] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2727.404360] LustreError: 81700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.456295] LustreError: 81700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2731.576114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.244727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2737.259129] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2746.005177] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2753.749733] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2761.560356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2761.871489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2762.954487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2769.209303] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2776.272381] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2787.812116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.842081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2792.932459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2792.934939] Lustre: Skipped 1 previous similar message [ 2793.327602] Lustre: server umount lustre-OST0000 complete [ 2798.060811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.083766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2798.088427] Lustre: Skipped 2 previous similar messages [ 2798.108062] Lustre: Skipped 1 previous similar message [ 2803.008665] Lustre: server umount lustre-MDT0000 complete [ 2803.168090] LustreError: 81700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2806.860192] LustreError: 81681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972949 with bad export cookie 13715064336595168923 [ 2806.871744] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2806.872483] LustreError: 81681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2807.262957] Lustre: server umount lustre-MDT0001 complete [ 2818.274522] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2821.678196] Key type lgssc unregistered [ 2822.108973] LNet: 85233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2822.122097] LNetError: 85233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2822.147607] LNet: Removed LNI 192.168.201.115@tcp [ 2823.272541] Key type .llcrypt unregistered [ 2823.275061] Key type ._llcrypt unregistered [ 2852.060362] Key type ._llcrypt registered [ 2852.061972] Key type .llcrypt registered [ 2852.143915] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2853.407078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2853.581081] alg: No test for adler32 (adler32-zlib) [ 2854.875252] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2855.338364] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2857.055235] Key type lgssc registered [ 2858.105548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2866.189176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2866.237722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.871744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.853083] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2878.435688] LustreError: 86642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2878.451027] LustreError: 86642:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2883.559494] LustreError: 86641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2884.209906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2884.945614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2890.967474] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2900.485748] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2906.762199] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2914.998810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2915.407415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.921434] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2932.345903] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2937.930227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2946.017288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2946.022868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.031585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2948.066168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2948.081945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2948.089112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2951.846387] Lustre: server umount lustre-OST0000 complete [ 2956.769129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2956.776481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.788507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2956.790492] Lustre: Skipped 1 previous similar message [ 2962.376291] Lustre: server umount lustre-MDT0000 complete [ 2963.430681] LustreError: 86641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.455799] LustreError: 86641:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2966.922463] LustreError: 86622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973109 with bad export cookie 3600713010599698486 [ 2966.941416] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2967.329861] Lustre: server umount lustre-MDT0001 complete [ 2980.332902] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2984.485966] Key type lgssc unregistered [ 2984.779598] LNet: 90197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2984.795744] LNetError: 90197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2984.809424] LNet: Removed LNI 192.168.201.115@tcp [ 2985.906558] Key type .llcrypt unregistered [ 2985.911575] Key type ._llcrypt unregistered [ 3020.642090] Key type ._llcrypt registered [ 3020.643930] Key type .llcrypt registered [ 3020.735747] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3021.497659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3021.576888] alg: No test for adler32 (adler32-zlib) [ 3022.621708] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 3023.023965] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3024.751260] Key type lgssc registered [ 3026.180596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3035.578704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3035.608655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3037.109847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.643109] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3047.400200] LustreError: 91613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.421291] LustreError: 91613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3052.292541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.517036] LustreError: 91614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.988150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3058.463202] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3067.059239] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3074.407814] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3081.967541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3082.210411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3088.421246] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3095.421669] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3106.809097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3125.220110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3125.228144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.300493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3127.266348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3127.274666] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3127.296681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.467866] Lustre: server umount lustre-OST0000 complete [ 3137.519417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.521664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3137.527054] Lustre: Skipped 1 previous similar message [ 3137.535524] Lustre: Skipped 1 previous similar message [ 3141.807502] Lustre: server umount lustre-MDT0000 complete [ 3142.626430] LustreError: 91613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.649235] LustreError: 91613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3146.647482] LustreError: 91593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973289 with bad export cookie 3446947143543521335 [ 3146.670564] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3146.673144] LustreError: 91593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3147.090900] Lustre: server umount lustre-MDT0001 complete [ 3160.111427] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3163.598763] Key type lgssc unregistered [ 3163.936682] LNet: 95185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3163.943177] LNetError: 95185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3163.976943] LNet: Removed LNI 192.168.201.115@tcp [ 3164.945686] Key type .llcrypt unregistered [ 3164.948346] Key type ._llcrypt unregistered [ 3193.302491] Key type ._llcrypt registered [ 3193.307519] Key type .llcrypt registered [ 3193.493482] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3194.816608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3195.013412] alg: No test for adler32 (adler32-zlib) [ 3196.189821] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 3196.545679] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3198.319225] Key type lgssc registered [ 3199.817659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3208.516983] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3208.535647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.095468] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3229.280871] LustreError: 96567:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [ 3255.968408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.817359] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3273.383592] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3279.617606] Lustre: server umount lustre-OST0000 complete [ 3289.213959] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 13:03:50 (1788973430) [ 3296.637053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.150259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3302.059156] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3302.377197] LustreError: 98208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3302.405469] LustreError: 98208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3307.499642] LustreError: 98208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3311.342463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.896801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3317.275278] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3325.262515] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3332.678577] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3339.423808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3339.600541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.686541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3348.780615] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3359.987879] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3372.513975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3372.517942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.527915] LustreError: Skipped 1 previous similar message [ 3372.528613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3372.587603] Lustre: Skipped 1 previous similar message [ 3377.634837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.638407] Lustre: Skipped 1 previous similar message [ 3378.586178] Lustre: server umount lustre-OST0000 complete [ 3383.776571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3383.796611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.820905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3383.831125] Lustre: Skipped 1 previous similar message [ 3387.874082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.879225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.713137] Lustre: server umount lustre-MDT0000 complete [ 3392.996408] LustreError: 98207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.018731] LustreError: 98207:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3393.945966] LustreError: 98189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973536 with bad export cookie 6372187175913875679 [ 3393.966581] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3394.274499] Lustre: server umount lustre-MDT0001 complete [ 3406.102283] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3408.448044] Key type lgssc unregistered [ 3408.662783] LNet: 101900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3408.666497] LNetError: 101900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3408.676838] LNet: Removed LNI 192.168.201.115@tcp [ 3409.637722] Key type .llcrypt unregistered [ 3409.639723] Key type ._llcrypt unregistered [ 3470.865397] Key type ._llcrypt registered [ 3470.868905] Key type .llcrypt registered [ 3471.009535] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3472.831749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3472.975673] alg: No test for adler32 (adler32-zlib) [ 3474.625712] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 3474.974938] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3476.743477] Key type lgssc registered [ 3478.522779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3489.890045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3489.928449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.633611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.554743] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3501.025823] LustreError: 104084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3501.056200] LustreError: 104084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3506.154950] LustreError: 104085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3507.387078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.951707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3513.585380] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3522.998497] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3530.324105] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3537.724553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3538.150325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3544.784740] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3552.721009] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3560.703066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3574.240361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3574.243224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3574.249196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3576.289667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3576.293380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3576.312817] Lustre: Skipped 1 previous similar message [ 3578.791552] Lustre: server umount lustre-OST0000 complete [ 3584.995191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3585.007087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3585.023489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3591.648947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3591.655859] Lustre: Skipped 2 previous similar messages [ 3596.771030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3596.783321] Lustre: Skipped 2 previous similar messages [ 3597.279452] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3597.525584] Lustre: server umount lustre-MDT0000 complete [ 3600.869354] LustreError: 105046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3600.893057] LustreError: 105046:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3601.860986] LustreError: 104065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973744 with bad export cookie 12211627560593736643 [ 3601.865814] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3601.879713] LustreError: 104065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3602.436231] Lustre: server umount lustre-MDT0001 complete [ 3614.576787] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3617.811706] Key type lgssc unregistered [ 3618.251693] LNet: 107643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3618.263538] LNetError: 107643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3618.301495] LNet: Removed LNI 192.168.201.115@tcp [ 3619.498594] Key type .llcrypt unregistered [ 3619.501933] Key type ._llcrypt unregistered [ 3641.727766] Key type ._llcrypt registered [ 3641.729781] Key type .llcrypt registered [ 3641.876842] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3643.096525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3643.111593] alg: No test for adler32 (adler32-zlib) [ 3644.443217] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 3644.668176] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3646.407318] Key type lgssc registered [ 3647.933080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3656.530815] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3656.557348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3672.096291] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3677.217226] LustreError: 108884:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [ 3703.904641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3709.758173] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3718.682703] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3726.145182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3726.494149] LustreError: MGC192.168.201.115@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3726.507836] LustreError: 109909:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3726.515844] LustreError: 109909:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3726.528642] LustreError: 109909:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3726.711413] Lustre: server umount lustre-MDT0000 complete [ 3726.713035] LustreError: 109909:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3745.946969] Lustre: server umount lustre-OST0000 complete [ 3756.762978] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 3768.104414] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3808.946796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.210897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3809.237085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3809.372808] Lustre: lustre-MDT0000: new disk, initializing [ 3809.533936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.559961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3814.679533] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3827.149856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3827.486904] Lustre: 112700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3827.617436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3827.625053] Lustre: Skipped 1 previous similar message [ 3827.717688] Lustre: lustre-MDT0001: new disk, initializing [ 3827.829901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3827.889779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3827.910102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3833.040144] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3841.753370] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3847.890562] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3857.085533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3857.351231] Lustre: lustre-OST0000: new disk, initializing [ 3857.355581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3857.362336] Lustre: 114669:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3857.500794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3864.712909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3864.730749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3864.912621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3866.554327] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3877.374860] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3882.056775] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3882.267591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3887.217224] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3887.424798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3890.660748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3890.665347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3890.669050] LustreError: Skipped 1 previous similar message [ 3890.672932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3890.699335] Lustre: Skipped 1 previous similar message [ 3895.701761] Lustre: server umount lustre-OST0000 complete [ 3900.900072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.915783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3900.918608] Lustre: Skipped 1 previous similar message [ 3900.940431] Lustre: Skipped 2 previous similar messages [ 3905.966281] Lustre: server umount lustre-MDT0000 complete [ 3909.830204] LustreError: 112694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974052 with bad export cookie 18255684784941574998 [ 3909.831374] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3909.838860] LustreError: 112694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3910.468542] Lustre: server umount lustre-MDT0001 complete [ 3919.427259] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 13:14:21 (1788974061) [ 3922.939722] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3940.249624] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 3949.996554] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3986.473332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.716959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3986.749924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3986.825479] Lustre: lustre-MDT0000: new disk, initializing [ 3986.903282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3986.918276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3991.468286] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4002.689246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.823299] Lustre: 119570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4002.913422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4002.919377] Lustre: Skipped 1 previous similar message [ 4003.003443] Lustre: lustre-MDT0001: new disk, initializing [ 4003.094322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4003.122983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4003.126197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4007.865987] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4018.096328] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4023.334367] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4030.582469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4030.870987] Lustre: lustre-OST0000: new disk, initializing [ 4030.874077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4030.879885] Lustre: 121537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4030.959037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4036.689661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4036.705203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4036.876984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4038.278537] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4047.977077] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4052.315930] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4052.489993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4057.298208] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4057.596689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4062.688608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4062.706267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4062.726495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4062.729810] Lustre: Skipped 1 previous similar message [ 4065.879881] Lustre: server umount lustre-OST0000 complete [ 4069.857516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4069.870122] LustreError: Skipped 1 previous similar message [ 4069.876389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.898876] Lustre: Skipped 1 previous similar message [ 4069.903239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4075.810230] Lustre: server umount lustre-MDT0000 complete [ 4078.071548] LustreError: 119581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.101884] LustreError: 119581:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4080.030859] LustreError: 119562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974222 with bad export cookie 18255684784941575880 [ 4080.040455] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4080.040682] LustreError: 119562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4080.651529] Lustre: server umount lustre-MDT0001 complete [ 4089.736752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.313645] LustreError: 123755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4090.374672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.465923] LustreError: 123756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4095.502279] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4100.577972] LustreError: 123755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4104.275429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.631970] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4116.616192] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4123.050845] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4130.863825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4138.233149] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4146.802706] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4157.023799] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4165.094971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4165.101503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4165.114788] Lustre: Skipped 2 previous similar messages [ 4165.131345] Lustre: Skipped 3 previous similar messages [ 4167.752271] Lustre: server umount lustre-OST0000 complete [ 4175.330548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.353295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4175.355792] Lustre: Skipped 1 previous similar message [ 4175.363569] Lustre: Skipped 1 previous similar message [ 4177.735949] Lustre: server umount lustre-MDT0000 complete [ 4180.449380] LustreError: 123755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4180.464357] LustreError: 123755:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4181.197945] LustreError: 123736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974323 with bad export cookie 18255684784941576762 [ 4181.200219] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4181.222469] LustreError: 123736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4181.655204] Lustre: server umount lustre-MDT0001 complete [ 4195.748922] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 4201.168272] Key type lgssc unregistered [ 4201.536060] LNet: 127383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4201.556744] LNetError: 127383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4201.607097] LNet: Removed LNI 192.168.201.115@tcp [ 4203.161716] Key type .llcrypt unregistered [ 4203.163306] Key type ._llcrypt unregistered [ 4224.038487] Key type ._llcrypt registered [ 4224.044504] Key type .llcrypt registered [ 4224.234915] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 4243.851421] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 4244.922409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4245.013210] alg: No test for adler32 (adler32-zlib) [ 4246.081924] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 4246.315249] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 4248.103286] Key type lgssc registered [ 4249.713897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4285.748411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4285.798164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.254730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4287.295272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4287.433179] Lustre: lustre-MDT0000: new disk, initializing [ 4287.539773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4287.595183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4293.484594] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4304.397492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.556543] Lustre: 130535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4304.633744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4304.642525] Lustre: Skipped 1 previous similar message [ 4304.748366] Lustre: lustre-MDT0001: new disk, initializing [ 4304.836367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4304.872404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4304.888882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4309.836748] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4320.086975] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4325.464491] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4331.991350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4332.242187] Lustre: lustre-OST0000: new disk, initializing [ 4332.245507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4332.251219] Lustre: 132505:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4332.383647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4339.860661] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4340.835104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4340.862778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4341.105379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4352.300792] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4358.586840] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4358.843670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4364.940435] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4365.215837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4371.938909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4371.945217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4371.952077] LustreError: Skipped 1 previous similar message [ 4371.956377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4371.973549] Lustre: Skipped 1 previous similar message [ 4371.999551] Lustre: Skipped 1 previous similar message [ 4373.724976] Lustre: server umount lustre-OST0000 complete [ 4381.666830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4381.674312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.697646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4384.212826] Lustre: server umount lustre-MDT0000 complete [ 4388.192798] LustreError: 130527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974530 with bad export cookie 4785041734895832928 [ 4388.202751] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4388.205245] LustreError: 130527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4388.493244] Lustre: server umount lustre-MDT0001 complete [ 4395.747267] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 13:22:17 (1788974537) [ 4401.723358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4402.096265] LustreError: 134917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.115669] LustreError: 134917:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4402.189583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4407.269375] LustreError: 134918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.680673] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4412.387168] LustreError: 134918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4416.559579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4417.067783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4422.288757] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4430.827413] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4437.404383] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4442.086923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4442.110689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4442.116242] Lustre: Skipped 2 previous similar messages [ 4442.142154] Lustre: Skipped 3 previous similar messages [ 4447.200669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4447.209806] Lustre: Skipped 1 previous similar message [ 4452.322665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4452.334888] Lustre: Skipped 1 previous similar message [ 4454.367165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4454.613295] Lustre: server umount lustre-MDT0000 complete [ 4457.440425] LustreError: 134918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.459678] LustreError: 134918:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4458.482520] LustreError: 135653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974600 with bad export cookie 4785041734895833810 [ 4458.496843] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4458.509930] LustreError: 135653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4458.792488] Lustre: server umount lustre-MDT0001 complete [ 4468.490269] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 13:23:29 (1788974609) [ 4476.272659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4491.809695] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4496.927738] LustreError: 137335:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [ 4522.657417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4529.736186] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4538.335962] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4540.805743] Lustre: server umount lustre-OST0000 complete [ 4550.330551] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 13:24:51 (1788974691) [ 4557.852586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.478381] LustreError: 138781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.685543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4564.261079] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4568.043186] LustreError: 138781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4568.060584] LustreError: 138781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4574.131078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.276976] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4588.157491] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4594.538161] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4602.177081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4602.507842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4602.514082] Lustre: Skipped 1 previous similar message [ 4609.710108] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4618.189438] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4629.472547] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4629.475817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4629.493807] Lustre: Skipped 1 previous similar message [ 4629.515790] Lustre: Skipped 2 previous similar messages [ 4632.389433] Lustre: server umount lustre-MDT0000 complete [ 4634.591962] LustreError: 138781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4634.611229] LustreError: 138781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4637.090484] LustreError: 138763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974779 with bad export cookie 4785041734895834475 [ 4637.095518] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4637.103263] LustreError: 138763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4637.619990] Lustre: server umount lustre-MDT0001 complete [ 4651.027416] Lustre: server umount lustre-OST0000 complete [ 4661.294616] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 13:26:42 (1788974802) [ 4669.468495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.337833] LustreError: 142437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4670.490630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.487588] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4686.553101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4691.777055] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4700.516231] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4708.705751] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4718.396121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4726.536610] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4737.207610] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4740.353159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4743.846535] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4744.209416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4750.883833] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4751.149737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4755.940989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4755.977184] Lustre: Skipped 1 previous similar message [ 4755.988035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4756.005352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4756.029705] Lustre: Skipped 1 previous similar message [ 4759.861675] Lustre: server umount lustre-OST0000 complete [ 4766.179730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.204360] Lustre: Skipped 1 previous similar message [ 4771.931570] Lustre: server umount lustre-MDT0000 complete [ 4776.420465] LustreError: 142432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4776.436363] LustreError: 142432:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4776.823383] LustreError: 142417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974919 with bad export cookie 4785041734895835546 [ 4776.829407] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4776.835150] LustreError: 142417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4777.090468] Lustre: server umount lustre-MDT0001 complete [ 4786.776468] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 13:28:48 (1788974928) [ 4795.022537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4810.783964] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4815.903456] LustreError: 146598:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [ 4841.695680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.704813] Lustre: Skipped 2 previous similar messages [ 4849.462542] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4857.932877] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4865.655701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4866.369086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4871.809988] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4881.814482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.774779] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4894.552286] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4901.173365] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4906.450869] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4906.655718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4911.941252] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4912.157560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4918.252031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4918.268377] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4918.271367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4918.271375] Lustre: Skipped 4 previous similar messages [ 4918.320056] Lustre: Skipped 2 previous similar messages [ 4920.435661] Lustre: server umount lustre-MDT0000 complete [ 4923.375882] LustreError: 147635:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4923.402160] LustreError: 147635:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4924.410332] LustreError: 146606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788975066 with bad export cookie 4785041734895836568 [ 4924.418715] LustreError: 146606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4924.428495] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4924.711267] Lustre: server umount lustre-MDT0001 complete [ 4939.110622] Lustre: server umount lustre-OST0000 complete [ 4949.799336] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 13:31:30 (1788975090) [ 4957.606414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4973.088950] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4978.207606] LustreError: 150795:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [ 5010.194640] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5017.717638] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5025.043898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5025.996339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5031.284551] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5040.732342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.235564] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5056.748423] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5063.763718] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5071.366533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5071.456700] Lustre: lustre-OST0001: new disk, initializing [ 5071.459259] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5071.464133] Lustre: 153670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5077.102988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5077.119692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5077.237360] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5078.436830] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5088.301532] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5092.909761] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5093.258449] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5097.433267] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5097.626928] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5102.562508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5102.562585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5102.571109] LustreError: Skipped 1 previous similar message [ 5102.572834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5102.572841] Lustre: Skipped 2 previous similar messages [ 5102.618302] Lustre: Skipped 2 previous similar messages [ 5105.498459] Lustre: server umount lustre-OST0000 complete [ 5113.312607] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5115.323889] Lustre: server umount lustre-OST0001 complete [ 5126.188469] Lustre: server umount lustre-MDT0000 complete [ 5131.578980] LustreError: 150802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788975273 with bad export cookie 4785041734895837520 [ 5131.585409] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5131.592672] LustreError: 150802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5132.149330] Lustre: server umount lustre-MDT0001 complete [ 5155.424751] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 13:34:56 (1788975296) [ 5157.185804] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5158.629376] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5160.429596] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 13:35:01 (1788975301) [ 5166.889662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5167.206453] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5167.243203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5167.538036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5167.546171] Lustre: Skipped 6 previous similar messages [ 5172.327126] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5182.150521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5182.214163] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5182.228615] Lustre: 157229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5188.074622] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5197.444445] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5203.482718] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5210.745404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5210.909223] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5218.338816] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5218.992535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5226.697687] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5232.355942] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5232.550267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5237.692733] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5237.906611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5244.392537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5244.405119] LustreError: Skipped 1 previous similar message [ 5244.413285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.435236] Lustre: Skipped 4 previous similar messages [ 5244.444735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5244.455600] Lustre: Skipped 7 previous similar messages [ 5246.572945] Lustre: server umount lustre-OST0000 complete [ 5270.346221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5277.644548] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5285.034643] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5285.096464] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.108713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.115887] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5290.767943] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5291.014866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5295.697858] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5295.858220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5300.786509] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff914aa0083800.ost_server_uuid 50 [ 5302.034121] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff914aa0083800.ost_server_uuid in FULL state after 0 sec [ 5302.155632] LustreError: 157235:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.15@tcp) returned error from blocking AST (req@00000000e63c134c x1875875608142848 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000006e46b250/0x4267e21b0be2aaaa lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.15@tcp remote: 0xab74cc74e74270ae expref: 5 pid: 157234 timeout: 5402 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5302.179627] LustreError: lustre-MDT0000: A client on nid 192.168.201.15@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5302.186792] LustreError: 157225:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.15@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000006e46b250/0x4267e21b0be2aaaa lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.15@tcp remote: 0xab74cc74e74270ae expref: 6 pid: 157234 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5307.875083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5307.883869] LustreError: Skipped 1 previous similar message [ 5312.935664] Lustre: server umount lustre-OST0000 complete [ 5323.393146] Lustre: server umount lustre-MDT0000 complete [ 5326.307094] LustreError: 157234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5326.319242] LustreError: 157234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 5327.477288] LustreError: 157220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788975469 with bad export cookie 4785041734895838696 [ 5327.483057] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5327.485781] LustreError: 157220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5333.944779] Lustre: server umount lustre-MDT0001 complete [ 5345.774654] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 5348.939971] Key type lgssc unregistered [ 5349.344706] LNet: 163344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5349.356995] LNetError: 163344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5349.377872] LNet: Removed LNI 192.168.201.115@tcp [ 5350.166830] Key type .llcrypt unregistered [ 5350.169526] Key type ._llcrypt unregistered [ 5375.637775] Key type ._llcrypt registered [ 5375.644188] Key type .llcrypt registered [ 5375.815364] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5376.976402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5377.217202] alg: No test for adler32 (adler32-zlib) [ 5378.285303] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 5378.688958] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 5380.384605] Key type lgssc registered [ 5382.145438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5390.846258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5390.868683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5392.443196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5397.660524] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5402.597218] LustreError: 164752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5402.625555] LustreError: 164752:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5407.551781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.715562] LustreError: 164751:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.005780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5412.441621] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5420.088360] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5425.988076] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5433.752990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5434.394561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5437.504271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5440.649910] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5448.870890] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5458.706598] Lustre: Failing over lustre-MDT0000 [ 5458.985656] Lustre: server umount lustre-MDT0000 complete [ 5459.427050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5459.441993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.476041] LustreError: 164751:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5463.012215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.012987] LustreError: 164752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5463.073379] LustreError: 164752:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5468.130207] LustreError: 164752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.159135] LustreError: 164752:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5471.809578] LustreError: 164734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.15@tcp arrived at 1788975614 with bad export cookie 17558555833884473821 [ 5471.840212] LustreError: 164734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5478.383533] LustreError: 165793:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5478.413368] LustreError: 165793:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5479.392264] Lustre: 164062:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788975605/real 1788975605] req@000000002ffc2342 x1875876795935616/t0(0) o400->MGC192.168.201.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788975621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5479.440095] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5481.464739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.982786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5490.017346] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5494.546780] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5495.277840] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5495.278996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5496.978104] Lustre: lustre-MDT0000: Denying connection for new client cbd25ab0-ae6d-40bd-8759-d0231f1b332f (at 192.168.201.15@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5505.510671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5505.521534] Lustre: Skipped 1 previous similar message [ 5505.527534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5505.534077] Lustre: Skipped 1 previous similar message [ 5510.629381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5510.635025] Lustre: Skipped 1 previous similar message [ 5511.572608] Lustre: server umount lustre-OST0000 complete [ 5516.167468] LustreError: 168491:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5516.186438] Lustre: 167836:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5516.194374] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5516.210996] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5516.220305] LustreError: 167836:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fc165646 x1875876795955712/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5516.234512] LustreError: 167836:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5516.245822] LustreError: 167836:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5516.254729] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5516.953747] Lustre: server umount lustre-MDT0000 complete [ 5520.864450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5520.884724] Lustre: Skipped 1 previous similar message [ 5520.895835] LustreError: 167192:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5520.919733] LustreError: 167192:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 5521.895271] LustreError: 164732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788975664 with bad export cookie 17558555833884473926 [ 5521.905479] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5521.923883] LustreError: 164732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5528.546601] Lustre: server umount lustre-MDT0001 complete [ 5543.715454] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 5547.091767] Key type lgssc unregistered [ 5547.709646] LNet: 169243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5547.716722] LNetError: 169243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5547.758995] LNet: Removed LNI 192.168.201.115@tcp [ 5549.209670] Key type .llcrypt unregistered [ 5549.214963] Key type ._llcrypt unregistered [ 5577.644621] Key type ._llcrypt registered [ 5577.651381] Key type .llcrypt registered [ 5577.756235] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5579.407492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5579.479385] alg: No test for adler32 (adler32-zlib) [ 5580.625043] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 5580.950474] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 5582.607366] Key type lgssc registered [ 5583.911823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5591.816603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5591.853614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5593.255388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5598.026448] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5602.796878] LustreError: 170653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5602.821494] LustreError: 170653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5606.955929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5607.424273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5611.592624] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5620.398498] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5627.836367] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5635.474590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5636.033884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5638.100169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5642.982853] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5650.571626] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5658.619428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5658.624208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.626102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5658.627215] LustreError: Skipped 1 previous similar message [ 5658.653238] Lustre: Skipped 1 previous similar message [ 5659.594252] Lustre: server umount lustre-OST0000 complete [ 5663.713232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5663.713644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5663.747746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5663.757118] Lustre: Skipped 2 previous similar messages [ 5668.833495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5668.843672] Lustre: Skipped 1 previous similar message [ 5669.666641] Lustre: server umount lustre-MDT0000 complete [ 5673.917898] LustreError: 172496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788975816 with bad export cookie 12537531517512721726 [ 5673.919476] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5673.942198] LustreError: 172496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5674.383127] Lustre: server umount lustre-MDT0001 complete [ 5685.208639] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 5687.840907] Key type lgssc unregistered [ 5688.189887] LNet: 174182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5688.198252] LNetError: 174182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5688.209730] LNet: Removed LNI 192.168.201.115@tcp [ 5689.082207] Key type .llcrypt unregistered [ 5689.086662] Key type ._llcrypt unregistered [ 5722.524190] Key type ._llcrypt registered [ 5722.526575] Key type .llcrypt registered [ 5722.608156] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5723.744071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5723.942526] alg: No test for adler32 (adler32-zlib) [ 5725.360098] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 5725.822947] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 5727.535396] Key type lgssc registered [ 5729.650775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5738.764177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5738.787427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.144252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5743.656184] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5749.734820] LustreError: 175879:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5749.752145] LustreError: 175879:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5753.020210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5753.398222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5758.410386] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5766.954595] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5774.235993] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5781.802593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5782.308677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.404686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5788.740541] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5796.886865] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5806.050070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5806.063758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5806.109304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5808.096352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5808.103674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5808.106287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.132678] Lustre: Skipped 1 previous similar message [ 5809.906986] Lustre: server umount lustre-OST0000 complete [ 5814.752435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5814.771802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5814.789716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5818.348233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5818.351025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5818.384851] Lustre: Skipped 1 previous similar message [ 5823.458835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5823.467471] Lustre: Skipped 1 previous similar message [ 5828.063720] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5828.172691] Lustre: server umount lustre-MDT0000 complete [ 5828.577083] LustreError: 175880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.603089] LustreError: 175880:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5831.992310] LustreError: 175860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788975974 with bad export cookie 5501045476412877510 [ 5832.001698] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5832.005611] LustreError: 175860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5832.460085] Lustre: server umount lustre-MDT0001 complete [ 5844.830166] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 5848.167810] Key type lgssc unregistered [ 5848.626848] LNet: 179417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5848.649448] LNetError: 179417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5848.675916] LNet: Removed LNI 192.168.201.115@tcp [ 5849.892920] Key type .llcrypt unregistered [ 5849.895759] Key type ._llcrypt unregistered [ 5877.217376] Key type ._llcrypt registered [ 5877.220596] Key type .llcrypt registered [ 5877.338369] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5878.816033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5878.849649] alg: No test for adler32 (adler32-zlib) [ 5879.934499] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 5880.220370] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 5881.951818] Key type lgssc registered [ 5883.572463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5895.885567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5895.929848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5897.497424] Lustre: *** cfs_fail_loc=135, val=0*** [ 5897.637213] LustreError: 180935:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5897.648436] LustreError: 180935:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.115@tcp: cfg command failed: rc = -2 [ 5897.663587] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5897.663587] [ 5897.673975] LustreError: MGC192.168.201.115@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5897.700708] LustreError: 180897:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5897.717089] LustreError: 180897:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5897.730866] LustreError: 180897:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5897.744497] LustreError: 180897:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5897.805667] Lustre: server umount lustre-MDT0000 complete [ 5897.811805] LustreError: 180897:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5905.857595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.509291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.191080] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5916.651068] LustreError: 181307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5916.670243] LustreError: 181307:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5920.528939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.329567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5927.738962] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5938.045825] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5944.353552] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5946.428065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.440056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5946.446216] Lustre: Skipped 1 previous similar message [ 5946.457933] Lustre: Skipped 1 previous similar message [ 5951.456446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5951.460677] Lustre: Skipped 1 previous similar message [ 5952.725245] Lustre: server umount lustre-MDT0000 complete [ 5956.368509] LustreError: 181286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976098 with bad export cookie 10914751710642102167 [ 5956.391734] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5956.766739] Lustre: server umount lustre-MDT0001 complete [ 5968.477381] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 5971.343493] Key type lgssc unregistered [ 5971.743515] LNet: 183625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5971.768883] LNetError: 183625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5971.792486] LNet: Removed LNI 192.168.201.115@tcp [ 5972.793517] Key type .llcrypt unregistered [ 5972.798794] Key type ._llcrypt unregistered [ 5995.900375] Key type ._llcrypt registered [ 5995.902802] Key type .llcrypt registered [ 5996.086784] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 5996.389396] Key type .llcrypt unregistered [ 5996.391056] Key type ._llcrypt unregistered [ 6016.562080] Key type ._llcrypt registered [ 6016.565526] Key type .llcrypt registered [ 6016.649818] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 6017.163994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6017.179358] alg: No test for adler32 (adler32-zlib) [ 6018.161476] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 6018.318930] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 6019.943860] Key type lgssc registered [ 6021.652911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6030.079116] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6030.119414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6045.855623] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6050.975457] LustreError: 185751:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [ 6077.664328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6083.853409] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6091.325177] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6099.216091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.306331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.468183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6105.975142] LustreError: 185778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.006434] LustreError: 185778:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6108.745303] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6111.212228] LustreError: 185777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6112.292355] LustreError: 186797:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6112.350110] LustreError: 186797:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6117.348016] LustreError: 186798:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6117.363513] LustreError: 186798:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6118.782461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6119.351935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6124.190487] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6135.991985] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6144.571152] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6148.096312] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6160.353813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6160.366213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.379360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.401829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.414200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.440748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6168.547846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6168.553367] Lustre: Skipped 2 previous similar messages [ 6173.667340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.679988] Lustre: Skipped 1 previous similar message [ 6174.175141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6174.296911] Lustre: server umount lustre-OST0000 complete [ 6178.785688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.801377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6178.803887] Lustre: Skipped 1 previous similar message [ 6184.335359] Lustre: server umount lustre-MDT0000 complete [ 6187.737507] LustreError: 185757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976330 with bad export cookie 7892078972866871322 [ 6187.755104] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6188.240570] Lustre: server umount lustre-MDT0001 complete [ 6199.219889] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 6202.146747] Key type lgssc unregistered [ 6202.520857] LNet: 189726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6202.533635] LNetError: 189726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6203.562322] LNet: Removed LNI 192.168.201.115@tcp [ 6204.509499] Key type .llcrypt unregistered [ 6204.512496] Key type ._llcrypt unregistered [ 6229.694867] Key type ._llcrypt registered [ 6229.697834] Key type .llcrypt registered [ 6229.811860] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 6231.068548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6231.224086] alg: No test for adler32 (adler32-zlib) [ 6232.309424] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 6232.610278] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 6234.266090] Key type lgssc registered [ 6235.638265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6244.696460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6244.739474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6246.342646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6251.294460] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6256.623435] LustreError: 191133:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6256.661856] LustreError: 191133:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6259.926793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6260.409287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6264.942140] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6272.736845] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6278.402152] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6285.698070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6286.041072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6291.184984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6291.854549] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6299.574322] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6305.173050] Lustre: Failing over lustre-MDT0000 [ 6305.447884] Lustre: server umount lustre-MDT0000 complete [ 6306.215161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6306.224109] LustreError: 191133:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6306.234762] Lustre: Skipped 1 previous similar message [ 6306.258612] LustreError: 191133:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6307.373993] LustreError: 192150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6311.395410] LustreError: 191130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6311.409848] LustreError: 191130:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6316.526751] LustreError: 191134:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.563639] LustreError: 191134:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6321.638383] Lustre: 190447:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788976448/real 1788976448] req@00000000c3531a88 x1875877690366848/t0(0) o400->MGC192.168.201.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788976464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6321.688450] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6324.464985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6326.760095] LustreError: 192150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6326.800251] LustreError: 192150:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 6331.872607] LustreError: 190443:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000007358e2c x1875877690373888/t0(0) o250->MGC192.168.201.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6332.092170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6332.166047] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6333.898104] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6337.076869] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6337.520265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6337.568970] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6337.645078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6344.075161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6349.574913] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6359.522949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6359.531404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6359.548817] Lustre: Skipped 1 previous similar message [ 6359.567101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6361.069430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6361.078454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6361.102413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6361.518603] Lustre: server umount lustre-OST0000 complete [ 6368.238108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6368.253917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6368.269000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6371.789368] Lustre: server umount lustre-MDT0000 complete [ 6373.351896] LustreError: 191134:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6373.393333] LustreError: 191134:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 6376.570691] LustreError: 191114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976518 with bad export cookie 7767426737866942228 [ 6376.580368] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6376.586164] LustreError: 191114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6377.071508] Lustre: server umount lustre-MDT0001 complete [ 6387.020321] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 6390.031323] Key type lgssc unregistered [ 6390.374772] LNet: 195940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6390.386976] LNetError: 195940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6391.397681] LNet: Removed LNI 192.168.201.115@tcp [ 6392.490122] Key type .llcrypt unregistered [ 6392.496147] Key type ._llcrypt unregistered [ 6420.506428] Key type ._llcrypt registered [ 6420.509635] Key type .llcrypt registered [ 6420.627371] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 6421.749550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6421.787056] alg: No test for adler32 (adler32-zlib) [ 6423.147981] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 6423.566603] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 6425.304470] Key type lgssc registered [ 6426.571685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6434.476064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6434.486937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6435.829669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.211543] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6446.055467] LustreError: 197352:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6446.078125] LustreError: 197352:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6450.684725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6451.073114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6455.416594] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6462.769181] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6468.691579] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6476.028136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6476.371233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6481.532162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6483.945508] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6491.601600] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6495.649463] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6502.106499] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6521.794201] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6532.070490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6532.081478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6532.097561] Lustre: Skipped 1 previous similar message [ 6532.105275] Lustre: Skipped 1 previous similar message [ 6534.991665] Lustre: server umount lustre-OST0000 complete [ 6542.309562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6542.313762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6542.333662] Lustre: Skipped 1 previous similar message [ 6545.223642] Lustre: server umount lustre-MDT0000 complete [ 6547.434638] LustreError: 197347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6547.446022] LustreError: 197347:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6548.610170] LustreError: 197332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976690 with bad export cookie 4790195544199033651 [ 6548.618070] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6548.630627] LustreError: 197332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6548.985145] Lustre: server umount lustre-MDT0001 complete [ 6557.716204] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 6560.640452] Key type lgssc unregistered [ 6561.054281] LNet: 201222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6561.056359] LNetError: 201222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6561.072137] LNet: Removed LNI 192.168.201.115@tcp [ 6562.045244] Key type .llcrypt unregistered [ 6562.049383] Key type ._llcrypt unregistered [ 6585.751253] Key type ._llcrypt registered [ 6585.757197] Key type .llcrypt registered [ 6585.945332] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 6587.427557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6587.446720] alg: No test for adler32 (adler32-zlib) [ 6588.619797] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 6588.822789] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 6590.529439] Key type lgssc registered [ 6591.713917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6598.803843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6598.820621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6600.484201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6604.659720] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6609.908265] LustreError: 202485:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6609.936147] LustreError: 202485:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6613.783338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6614.325966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6621.775679] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6633.297612] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6640.514443] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6647.231130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.566970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6654.598548] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6662.414677] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6670.126944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6678.004126] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6690.491458] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6705.741835] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6712.294817] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6725.600481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6725.611570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.628784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6726.627442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6726.628242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6726.646066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6731.744995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6731.753733] Lustre: Skipped 2 previous similar messages [ 6736.867797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6736.878185] Lustre: Skipped 1 previous similar message [ 6738.399168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6738.576091] Lustre: server umount lustre-OST0000 complete [ 6747.105750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6747.132518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6747.141511] Lustre: Skipped 1 previous similar message [ 6749.093456] Lustre: server umount lustre-MDT0000 complete [ 6752.230422] LustreError: 202481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6752.251111] LustreError: 202481:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6752.892105] LustreError: 206469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976895 with bad export cookie 15282104898596681044 [ 6752.902295] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6752.909185] LustreError: 206469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6753.229613] Lustre: server umount lustre-MDT0001 complete [ 6764.306323] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 6767.139935] Key type lgssc unregistered [ 6767.441830] LNet: 207734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6767.450583] LNetError: 207734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6767.484738] LNet: Removed LNI 192.168.201.115@tcp [ 6768.406862] Key type .llcrypt unregistered [ 6768.410781] Key type ._llcrypt unregistered [ 6793.946190] Key type ._llcrypt registered [ 6793.949192] Key type .llcrypt registered [ 6794.131813] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 6795.587567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6795.689232] alg: No test for adler32 (adler32-zlib) [ 6797.006822] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 6797.409751] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 6799.183286] Key type lgssc registered [ 6800.318199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6808.448982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6808.483375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6809.945310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6814.142938] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6820.332209] LustreError: 209145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6820.380220] LustreError: 209145:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6822.535347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6822.898638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6827.149609] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6834.367591] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6840.258346] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6847.117325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6847.623478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6853.550228] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6854.778050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6861.314452] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6875.623635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6875.627923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6875.636956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6875.656651] Lustre: Skipped 1 previous similar message [ 6880.740202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6880.751558] Lustre: Skipped 1 previous similar message [ 6881.243563] Lustre: server umount lustre-OST0000 complete [ 6885.858699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6885.875376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6885.881503] Lustre: Skipped 2 previous similar messages [ 6885.901407] Lustre: Skipped 1 previous similar message [ 6890.977793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6890.989631] Lustre: Skipped 1 previous similar message [ 6891.547848] Lustre: server umount lustre-MDT0000 complete [ 6895.362746] LustreError: 209127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977037 with bad export cookie 3428999267514910958 [ 6895.365296] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6895.378717] LustreError: 209127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6896.020412] Lustre: server umount lustre-MDT0001 complete [ 6905.040732] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 6908.088773] Key type lgssc unregistered [ 6908.529696] LNet: 212829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6908.534184] LNetError: 212829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6908.548303] LNet: Removed LNI 192.168.201.115@tcp [ 6909.483552] Key type .llcrypt unregistered [ 6909.485275] Key type ._llcrypt unregistered [ 6933.523184] Key type ._llcrypt registered [ 6933.525304] Key type .llcrypt registered [ 6933.653617] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 6935.393087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6935.471302] alg: No test for adler32 (adler32-zlib) [ 6936.682539] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 6936.868035] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 6938.521743] Key type lgssc registered [ 6939.658573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6946.875733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6946.902813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6948.248166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6952.672340] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6958.569646] LustreError: 214093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6958.603207] LustreError: 214093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6961.465812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6961.969828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6966.916403] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6974.583299] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6979.410673] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6987.860717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6988.161991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6993.294156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6997.201561] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7006.192559] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7023.080105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7023.087389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7023.104960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7025.917530] Lustre: server umount lustre-OST0000 complete [ 7033.319175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7033.326766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7033.349054] Lustre: Skipped 2 previous similar messages [ 7033.385653] Lustre: Skipped 2 previous similar messages [ 7036.135522] Lustre: server umount lustre-MDT0000 complete [ 7038.448180] LustreError: 214094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7038.483251] LustreError: 214094:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7040.166582] LustreError: 217025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977182 with bad export cookie 5343986214376410847 [ 7040.182830] LustreError: 217025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7040.191581] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7040.966751] Lustre: server umount lustre-MDT0001 complete [ 7053.077137] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7055.424564] Key type lgssc unregistered [ 7055.658866] LNet: 217777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7055.666032] LNetError: 217777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7056.741164] LNet: Removed LNI 192.168.201.115@tcp [ 7057.471758] Key type .llcrypt unregistered [ 7057.473367] Key type ._llcrypt unregistered [ 7077.196809] Key type ._llcrypt registered [ 7077.201652] Key type .llcrypt registered [ 7077.348315] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7079.058688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7079.103456] alg: No test for adler32 (adler32-zlib) [ 7080.482925] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 7080.835104] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7082.604287] Key type lgssc registered [ 7084.103766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7092.464612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7092.488402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7094.011668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7098.764789] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7104.480145] LustreError: 219042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7104.497543] LustreError: 219042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7108.451522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7109.011562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7114.073914] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7121.527404] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7127.676755] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7135.038722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7135.528853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7139.668285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7143.195241] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7150.684146] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7159.751576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7159.836266] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7160.113685] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7166.842461] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7175.722876] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7177.284523] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7189.231501] Lustre: Permanently deactivating lustre-OST0001 [ 7189.249597] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7196.283652] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7203.111664] Lustre: Permanently reactivating lustre-OST0001 [ 7203.130687] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7203.137465] Lustre: Skipped 2 previous similar messages [ 7205.564706] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7205.571617] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7205.577043] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7205.589256] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7205.642838] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7218.629355] Lustre: server umount lustre-OST0001 complete [ 7221.218084] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7221.230254] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7221.244776] LustreError: 221890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7221.301474] Lustre: Skipped 2 previous similar messages [ 7225.825489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7225.838973] LustreError: Skipped 1 previous similar message [ 7225.849092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7225.868569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7226.356989] LustreError: 221890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7226.387973] LustreError: 221890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7229.168224] Lustre: server umount lustre-OST0000 complete [ 7236.589214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.604263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7236.611177] Lustre: Skipped 2 previous similar messages [ 7236.634110] Lustre: Skipped 3 previous similar messages [ 7238.974497] Lustre: server umount lustre-MDT0000 complete [ 7241.699191] LustreError: 219771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7241.722306] LustreError: 219771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7242.807189] LustreError: 219024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977385 with bad export cookie 10693426432652008335 [ 7242.808419] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7242.832313] LustreError: 219024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7243.570767] Lustre: server umount lustre-MDT0001 complete [ 7254.488276] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7257.467768] Key type lgssc unregistered [ 7257.829712] LNet: 224110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7257.840975] LNetError: 224110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7258.853807] LNet: Removed LNI 192.168.201.115@tcp [ 7259.766780] Key type .llcrypt unregistered [ 7259.769241] Key type ._llcrypt unregistered [ 7300.269083] Key type ._llcrypt registered [ 7300.270525] Key type .llcrypt registered [ 7300.418828] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7301.862508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7301.929870] alg: No test for adler32 (adler32-zlib) [ 7303.129556] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 7303.361308] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7305.023213] Key type lgssc registered [ 7306.344756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7315.210603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7315.244037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.488475] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7316.536880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7316.763861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7321.455439] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7330.341520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.399978] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7330.418264] Lustre: 226315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7330.603678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7335.214291] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7343.288579] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7350.110628] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7357.030555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7357.283075] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7357.656984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7364.686460] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7366.288208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7372.989704] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7376.922732] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7380.235905] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7385.894538] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7396.355359] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7410.814890] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7410.824225] Lustre: Skipped 1 previous similar message [ 7432.411139] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7432.422037] Lustre: Skipped 2 previous similar messages [ 7469.796562] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7469.805589] Lustre: Skipped 4 previous similar messages [ 7535.182280] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7535.188365] Lustre: Skipped 7 previous similar messages [ 7545.325438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7545.336192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.356924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.838519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.844613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7549.408708] Lustre: server umount lustre-OST0000 complete [ 7556.069320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7556.076909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7556.088749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7556.097499] Lustre: Skipped 1 previous similar message [ 7556.112436] Lustre: Skipped 2 previous similar messages [ 7559.452373] Lustre: server umount lustre-MDT0000 complete [ 7563.478277] LustreError: 226305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977705 with bad export cookie 4432397759290881734 [ 7563.490478] LustreError: 226305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7563.494119] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7563.820584] Lustre: server umount lustre-MDT0001 complete [ 7576.419713] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7579.257175] Key type lgssc unregistered [ 7579.599284] LNet: 231108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7579.606666] LNetError: 231108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7580.647550] LNet: Removed LNI 192.168.201.115@tcp [ 7581.575716] Key type .llcrypt unregistered [ 7581.577088] Key type ._llcrypt unregistered [ 7610.077495] Key type ._llcrypt registered [ 7610.079431] Key type .llcrypt registered [ 7610.268645] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7612.637087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7612.747789] alg: No test for adler32 (adler32-zlib) [ 7614.052452] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 7614.470423] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7616.247279] Key type lgssc registered [ 7618.243348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7628.907772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7628.938037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7630.633786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7636.277741] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7640.057782] LustreError: 232520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7640.086694] LustreError: 232520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7645.153389] LustreError: 232519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7646.551687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7646.903855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7652.616960] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7661.190690] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7667.772392] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7675.349692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7675.791662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7683.136509] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7691.588196] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7700.351847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7720.931448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7720.940150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7720.943673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7725.937139] Lustre: server umount lustre-OST0000 complete [ 7731.170577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7731.177081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7731.195177] Lustre: Skipped 2 previous similar messages [ 7731.214576] Lustre: Skipped 2 previous similar messages [ 7736.290040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7736.304639] Lustre: Skipped 1 previous similar message [ 7736.887414] Lustre: server umount lustre-MDT0000 complete [ 7740.899761] LustreError: 232501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977883 with bad export cookie 1920001248541293957 [ 7740.901737] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7740.906730] LustreError: 232501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7741.319717] Lustre: server umount lustre-MDT0001 complete [ 7753.351629] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7756.093931] Key type lgssc unregistered [ 7756.381718] LNet: 236233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7756.390716] LNetError: 236233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7756.412288] LNet: Removed LNI 192.168.201.115@tcp [ 7757.422674] Key type .llcrypt unregistered [ 7757.430027] Key type ._llcrypt unregistered [ 7786.152987] Key type ._llcrypt registered [ 7786.161381] Key type .llcrypt registered [ 7786.373398] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7789.038116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7789.083858] alg: No test for adler32 (adler32-zlib) [ 7790.381834] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 7790.775810] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7792.679206] Key type lgssc registered [ 7795.494647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8069.580518] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 8074.218498] Key type lgssc unregistered [ 8074.564419] LNet: 238252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8074.567881] LNetError: 238252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8074.577962] LNet: Removed LNI 192.168.201.115@tcp [ 8075.434627] Key type .llcrypt unregistered [ 8075.438729] Key type ._llcrypt unregistered [ 8104.440448] Key type ._llcrypt registered [ 8104.442182] Key type .llcrypt registered [ 8104.554930] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 8123.600513] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8124.735220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8124.869523] alg: No test for adler32 (adler32-zlib) [ 8126.003556] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 8126.277892] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 8127.975215] Key type lgssc registered [ 8129.146696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8161.949089] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7826 sec ========== 14:25:03 (1788978303) [ 8163.746908] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:25:05 (1788978305) === [ 8167.294625] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:25:08 (1788978308) === [ 8211.215192] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 8215.414139] Key type lgssc unregistered [ 8215.886698] LNet: 243487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8215.899471] LNetError: 243487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8215.928988] LNet: Removed LNI 192.168.201.115@tcp [ 8217.209540] Key type .llcrypt unregistered [ 8217.211396] Key type ._llcrypt unregistered