[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 536403401 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002284] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.005000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.005000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.005025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.006012] pid_max: default: 32768 minimum: 301 [ 0.007161] LSM: Security Framework initializing [ 0.008069] Yama: becoming mindful. [ 0.009040] SELinux: Initializing. [ 0.010075] *** VALIDATE selinux *** [ 0.019199] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024116] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026114] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029075] *** VALIDATE tmpfs *** [ 0.030359] *** VALIDATE proc *** [ 0.031199] *** VALIDATE cgroup *** [ 0.032005] *** VALIDATE cgroup2 *** [ 0.033221] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034132] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036020] Spectre V2 : User space: Vulnerable [ 0.037004] Speculative Store Bypass: Vulnerable [ 0.039636] debug: unmapping init [mem 0xffffffff9e059000-0xffffffff9e060fff] [ 0.041834] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042707] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043020] ... version: 2 [ 0.044011] ... bit width: 48 [ 0.045016] ... generic registers: 4 [ 0.046010] ... value mask: 0000ffffffffffff [ 0.047014] ... max period: 00007fffffffffff [ 0.048010] ... fixed-purpose events: 3 [ 0.049011] ... event mask: 000000070000000f [ 0.050336] rcu: Hierarchical SRCU implementation. [ 0.052463] smp: Bringing up secondary CPUs ... [ 0.053660] x86: Booting SMP configuration: [ 0.054019] .... node #0, CPUs: #1 #2 #3 [ 0.058065] smp: Brought up 1 node, 4 CPUs [ 0.060039] smpboot: Max logical packages: 1 [ 0.061000] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.117054] node 0 deferred pages initialised in 54ms [ 0.121111] devtmpfs: initialized [ 0.122271] x86/mm: Memory block size: 128MB [ 0.126000] gcov: version magic: 0x41383552 [ 0.129507] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.132284] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.135601] pinctrl core: initialized pinctrl subsystem [ 0.137256] [ 0.137842] ************************************************************* [ 0.140013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142012] ** ** [ 0.144016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147016] ** ** [ 0.149015] ** This means that this kernel is built to expose internal ** [ 0.151011] ** IOMMU data structures, which may compromise security on ** [ 0.152021] ** your system. ** [ 0.153000] ** ** [ 0.153026] ** If you see this message and you are not debugging the ** [ 0.156031] ** kernel, report this immediately to your vendor! ** [ 0.159019] ** ** [ 0.162023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164022] ************************************************************* [ 0.167906] NET: Registered protocol family 16 [ 0.170755] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174122] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178153] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182117] cpuidle: using governor menu [ 0.183957] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186548] PCI: Using configuration type 1 for base access [ 0.189120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.199356] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206055] cryptd: max_cpu_qlen set to 1000 [ 0.210282] ACPI: Added _OSI(Module Device) [ 0.213036] ACPI: Added _OSI(Processor Device) [ 0.216020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.225554] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.232350] ACPI: Interpreter enabled [ 0.233058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.235012] ACPI: Using IOAPIC for interrupt routing [ 0.237150] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.240432] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.251802] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.254036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.256020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260529] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266446] acpiphp: Slot [2] registered [ 0.269213] acpiphp: Slot [5] registered [ 0.274146] acpiphp: Slot [6] registered [ 0.275088] acpiphp: Slot [7] registered [ 0.276170] acpiphp: Slot [8] registered [ 0.278194] acpiphp: Slot [9] registered [ 0.280111] acpiphp: Slot [10] registered [ 0.281107] acpiphp: Slot [3] registered [ 0.283130] acpiphp: Slot [4] registered [ 0.284104] acpiphp: Slot [11] registered [ 0.286160] acpiphp: Slot [12] registered [ 0.288095] acpiphp: Slot [13] registered [ 0.289221] acpiphp: Slot [14] registered [ 0.291109] acpiphp: Slot [15] registered [ 0.293182] acpiphp: Slot [16] registered [ 0.295139] acpiphp: Slot [17] registered [ 0.297111] acpiphp: Slot [18] registered [ 0.298290] acpiphp: Slot [19] registered [ 0.300107] acpiphp: Slot [20] registered [ 0.302317] acpiphp: Slot [21] registered [ 0.304123] acpiphp: Slot [22] registered [ 0.306166] acpiphp: Slot [23] registered [ 0.307153] acpiphp: Slot [24] registered [ 0.309107] acpiphp: Slot [25] registered [ 0.311179] acpiphp: Slot [26] registered [ 0.313229] acpiphp: Slot [27] registered [ 0.315135] acpiphp: Slot [28] registered [ 0.317132] acpiphp: Slot [29] registered [ 0.319116] acpiphp: Slot [30] registered [ 0.320154] acpiphp: Slot [31] registered [ 0.322049] PCI host bridge to bus 0000:00 [ 0.324031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.326021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.328018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.331027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333047] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.337032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.343445] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.349573] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.352037] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.354011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.361323] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.363810] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.366043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.369958] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.375016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.388103] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.390023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.396261] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.402027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.408027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.431778] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.438013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.444018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.454013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.460108] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.464014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.467013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.477017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.486104] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.493016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.499024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.511020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.517601] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.526017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.531014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.544017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.557573] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.566022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.574017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.591020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.608484] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.612842] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.615392] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.617374] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.619219] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.625048] iommu: Default domain type: Passthrough [ 0.626421] SCSI subsystem initialized [ 0.628110] ACPI: bus type USB registered [ 0.629117] usbcore: registered new interface driver usbfs [ 0.631048] usbcore: registered new interface driver hub [ 0.632050] usbcore: registered new device driver usb [ 0.633095] pps_core: LinuxPPS API ver. 1 registered [ 0.634006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.636048] PTP clock support registered [ 0.637109] EDAC MC: Ver: 3.0.0 [ 0.638291] PCI: Using ACPI for IRQ routing [ 0.639601] NetLabel: Initializing [ 0.641008] NetLabel: domain hash size = 128 [ 0.641911] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.642103] NetLabel: unlabeled traffic allowed by default [ 0.644120] vgaarb: loaded [ 0.647307] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.648009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.655932] clocksource: Switched to clocksource kvm-clock [ 0.839170] VFS: Disk quotas dquot_6.6.0 [ 0.842471] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.847678] *** VALIDATE ramfs *** [ 0.849311] *** VALIDATE hugetlbfs *** [ 0.851742] pnp: PnP ACPI init [ 0.856691] pnp: PnP ACPI: found 6 devices [ 0.878830] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.881330] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.882810] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.884461] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.886213] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.887846] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.889903] NET: Registered protocol family 2 [ 0.891853] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.896134] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.898470] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.903060] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.905672] TCP: Hash tables configured (established 65536 bind 65536) [ 0.907843] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.910156] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.912359] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.914840] NET: Registered protocol family 1 [ 0.916776] RPC: Registered named UNIX socket transport module. [ 0.918132] RPC: Registered udp transport module. [ 0.919158] RPC: Registered tcp transport module. [ 0.921394] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.923097] NET: Registered protocol family 44 [ 0.924385] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.925792] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.927323] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.929104] PCI: CLS 0 bytes, default 64 [ 0.930376] Unpacking initramfs... [ 2.425411] debug: unmapping init [mem 0xffff9f307cc54000-0xffff9f307ffbffff] [ 2.429576] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.431583] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.434051] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.913167] Initialise system trusted keyrings [ 2.914347] Key type blacklist registered [ 2.915938] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.925372] zbud: loaded [ 2.928303] *** VALIDATE nfs *** [ 2.929474] *** VALIDATE nfs4 *** [ 2.931031] pstore: using deflate compression [ 2.934423] Platform Keyring initialized [ 3.028493] NET: Registered protocol family 38 [ 3.030201] Key type asymmetric registered [ 3.031801] Asymmetric key parser 'x509' registered [ 3.033602] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.036892] io scheduler mq-deadline registered [ 3.038642] io scheduler kyber registered [ 3.040461] io scheduler bfq registered [ 3.042165] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.044965] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.047883] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.051084] ACPI: Power Button [PWRF] [ 3.056379] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.062742] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.077254] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.086150] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.101438] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.128983] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.156026] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.160816] Non-volatile memory driver v1.3 [ 3.162553] Linux agpgart interface v0.103 [ 3.193753] virtio_blk virtio1: [vda] 149992 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.196758] vda: detected capacity change from 0 to 76795904 [ 3.213660] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.216309] vdb: detected capacity change from 0 to 1073741824 [ 3.234553] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.237732] vdc: detected capacity change from 0 to 2621440000 [ 3.441365] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.444558] vdd: detected capacity change from 0 to 2621440000 [ 3.459150] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.462097] vde: detected capacity change from 0 to 4294967296 [ 3.487578] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.490571] vdf: detected capacity change from 0 to 4294967296 [ 3.740492] libphy: Fixed MDIO Bus: probed [ 3.772651] usbcore: registered new interface driver usbserial_generic [ 3.775847] usbserial: USB Serial support registered for generic [ 3.777638] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.781549] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.782922] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.022638] mousedev: PS/2 mouse device common for all mice [ 4.025716] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.026626] rtc_cmos 00:05: RTC can wake from S4 [ 4.034098] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.034399] rtc_cmos 00:05: registered as rtc0 [ 4.042796] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.045854] intel_pstate: CPU model not supported [ 4.049142] hid: raw HID events driver (C) Jiri Kosina [ 4.051526] usbcore: registered new interface driver usbhid [ 4.054109] usbhid: USB HID core driver [ 4.054367] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.057082] drop_monitor: Initializing network drop monitor service [ 4.057211] Initializing XFRM netlink socket [ 4.057560] NET: Registered protocol family 10 [ 4.064664] Segment Routing with IPv6 [ 4.070466] NET: Registered protocol family 17 [ 4.072435] mpls_gso: MPLS GSO support [ 4.077383] RAS: Correctable Errors collector initialized. [ 4.079601] AVX version of gcm_enc/dec engaged. [ 4.081524] AES CTR mode by8 optimization enabled [ 4.169638] sched_clock: Marking stable (4169555109, 0)->(5069316297, -899761188) [ 4.173669] registered taskstats version 1 [ 4.175527] Loading compiled-in X.509 certificates [ 4.177745] zswap: loaded using pool lzo/zbud [ 4.206659] Key type big_key registered [ 4.221358] Key type encrypted registered [ 4.222730] ima: No TPM chip found, activating TPM-bypass! [ 4.224520] ima: Allocated hash algorithm: sha1 [ 4.226763] ima: No architecture policies found [ 4.228678] evm: Initialising EVM extended attributes: [ 4.230670] evm: security.selinux [ 4.231818] evm: security.ima [ 4.232976] evm: security.capability [ 4.234422] evm: HMAC attrs: 0x1 [ 4.236809] rtc_cmos 00:05: setting system clock to 2026-09-26 02:49:25 UTC (1790390965) [ 4.244254] debug: unmapping init [mem 0xffffffff9f003000-0xffffffff9f1fffff] [ 4.247569] debug: unmapping init [mem 0xffffffff9dd82000-0xffffffff9e058fff] [ 4.257169] Write protecting the kernel read-only data: 28672k [ 4.260858] debug: unmapping init [mem 0xffffffff9c403000-0xffffffff9c5fffff] [ 4.263221] debug: unmapping init [mem 0xffffffff9cd14000-0xffffffff9cdfffff] [ 4.301564] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.312907] systemd[1]: Detected virtualization kvm. [ 4.316338] systemd[1]: Detected architecture x86-64. [ 4.319283] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.346413] systemd[1]: No hostname configured. [ 4.348735] systemd[1]: Set hostname to . [ 4.350679] random: systemd: uninitialized urandom read (16 bytes read) [ 4.353663] systemd[1]: Initializing machine ID from random generator. [ 4.562339] random: systemd: uninitialized urandom read (16 bytes read) [ 4.565413] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.577660] random: systemd: uninitialized urandom read (16 bytes read) [ 4.580460] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 4.587948] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.442649] device-mapper: uevent: version 1.0.3 [ 5.444350] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. [ OK [ 6.124595] random: fast init done ] Mounted Kernel Configuration File System. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 6.361807] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.563836] scsi host0: ata_piix [ 6.615312] scsi host1: ata_piix [ 6.619024] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.623208] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.756751] random: crng init done [ 11.764716] random: 7 urandom warning(s) missed due to ratelimiting [ 11.868801] dracut-initqueue[568]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.385627] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.295526] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.167258] SELinux: Disabled at runtime. [ 18.285104] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.308125] systemd[1]: Detected virtualization kvm. [ 18.310837] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.878146] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.884631] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.901105] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.911106] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.926582] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.940466] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.023781] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ 20.308345] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.545569] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.653195] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.011571] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.644398] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.733133] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 29.883346] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 30.955430] NFS: Registering the id_resolver key type [ 30.969637] Key type id_resolver registered [ 30.981976] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg119-server login: [ 68.952612] hrtimer: interrupt took 6795944 ns [ 91.549289] libcfs: loading out-of-tree module taints kernel. [ 91.670678] Key type ._llcrypt registered [ 91.672106] Key type .llcrypt registered [ 91.879937] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 112.364363] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 114.235627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.258895] alg: No test for adler32 (adler32-zlib) [ 115.837636] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 116.686268] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 118.599171] Key type lgssc registered [ 120.661706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 140.101820] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 181.580606] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 194.930971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 194.972876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.278742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.299495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.388191] Lustre: lustre-MDT0000: new disk, initializing [ 196.458146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.470202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.198931] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.689745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.878986] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.931817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 215.934823] Lustre: Skipped 1 previous similar message [ 216.071406] Lustre: lustre-MDT0001: new disk, initializing [ 216.166121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.225114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.233216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 220.667915] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.697486] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 236.163405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.564685] Lustre: lustre-OST0000: new disk, initializing [ 236.569995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 236.576744] Lustre: 8440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.639652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 243.294611] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.838934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 246.859175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 246.931261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 257.888035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.008835] Lustre: lustre-OST0001: new disk, initializing [ 258.010869] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 258.013828] Lustre: 9513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.049048] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 264.351862] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.309931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 266.314436] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 266.460354] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.472436] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 284.852505] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 291.496485] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing check_logdir /tmp/testlogs/ [ 297.251780] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing yml_node [ 301.878516] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 304.617748] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 307.587396] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 309.647650] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 25 22:54:29 EDT 2026 [ 328.700051] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 330.254470] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 348.648797] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.659285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 348.666658] Lustre: Skipped 3 previous similar messages [ 348.708722] Lustre: Skipped 3 previous similar messages [ 352.242835] Lustre: server umount lustre-MDT0000 complete [ 356.520075] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790391317 with bad export cookie 1663811897608147567 [ 356.536251] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 356.540866] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.889540] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.901526] Lustre: Skipped 1 previous similar message [ 358.902374] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 358.902530] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 358.939792] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 361.888118] LustreError: 6506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 361.888833] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 361.906194] LustreError: 6506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 361.908747] Lustre: Skipped 2 previous similar messages [ 362.939788] Lustre: server umount lustre-MDT0001 complete [ 367.314758] Lustre: server umount lustre-OST0000 complete [ 371.594992] Lustre: server umount lustre-OST0001 complete [ 378.842625] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 389.578560] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 426.237354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.556884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 426.607739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 426.708418] Lustre: lustre-MDT0000: new disk, initializing [ 426.825658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.840939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 431.509231] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 441.412414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.537982] Lustre: 15069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 441.604801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 441.608313] Lustre: Skipped 1 previous similar message [ 441.700799] Lustre: lustre-MDT0001: new disk, initializing [ 441.787612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 441.849808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 441.870561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 448.383523] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 459.870758] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 465.866642] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 473.398452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 473.784871] Lustre: lustre-OST0000: new disk, initializing [ 473.793395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 473.803329] Lustre: 17039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 473.928308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 475.082784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 475.092907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 475.202874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 481.729410] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 493.088029] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 498.490791] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 498.796556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 503.232477] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 503.463237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 505.825730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 505.840073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.841915] LustreError: Skipped 1 previous similar message [ 505.874070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.946404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.950380] Lustre: Skipped 2 previous similar messages [ 511.772667] Lustre: server umount lustre-OST0000 complete [ 516.067164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.088415] Lustre: Skipped 2 previous similar messages [ 521.185590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 521.200489] Lustre: Skipped 3 previous similar messages [ 530.399719] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 530.666886] Lustre: server umount lustre-MDT0000 complete [ 531.428608] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.458803] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 535.356565] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790391496 with bad export cookie 1663811897608149100 [ 535.375644] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 535.378022] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.728856] Lustre: server umount lustre-MDT0001 complete [ 539.974685] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:58:19 (1790391499) [ 547.551826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.114952] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.274374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.417374] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 553.443210] LustreError: 19307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.414418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.526183] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 576.388578] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 583.562184] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 590.279595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 597.492714] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 604.993839] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 617.961746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 617.967722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.981870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 617.985326] Lustre: Skipped 2 previous similar messages [ 621.895746] Lustre: server umount lustre-OST0000 complete [ 628.714062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.723165] Lustre: Skipped 1 previous similar message [ 631.916192] Lustre: server umount lustre-MDT0000 complete [ 633.825471] LustreError: 19302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.859793] LustreError: 19302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 637.301653] LustreError: 19286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790391598 with bad export cookie 1663811897608149982 [ 637.309261] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.314849] LustreError: 19286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 637.855718] Lustre: server umount lustre-MDT0001 complete [ 651.224753] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 653.975730] Key type lgssc unregistered [ 654.195636] LNet: 22846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.207226] LNetError: 22846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.221753] LNet: Removed LNI 192.168.201.119@tcp [ 655.639134] Key type .llcrypt unregistered [ 655.641041] Key type ._llcrypt unregistered [ 684.119788] Key type ._llcrypt registered [ 684.124644] Key type .llcrypt registered [ 684.270598] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 685.548218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 685.885135] alg: No test for adler32 (adler32-zlib) [ 686.907366] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 687.117209] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 688.807417] Key type lgssc registered [ 689.890593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 697.220701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 697.234164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.544540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 703.119885] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 708.065477] LustreError: 24255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.091017] LustreError: 24255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 711.748665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.243947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 717.597232] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 725.284882] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 730.897941] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 737.051222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 737.494484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 743.691425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 743.824473] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 750.812413] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 768.765835] Lustre: server umount lustre-OST0000 complete [ 773.600540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 773.610229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.637227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 774.624761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 774.626239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.748896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.758770] Lustre: Skipped 2 previous similar messages [ 782.915861] Lustre: server umount lustre-MDT0000 complete [ 784.865456] Lustre: 23566:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790391730/real 1790391730] req@0000000064cdd858 x1877361712723840/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790391746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 784.870192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.889473] LustreError: 24279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 784.894412] Lustre: 23566:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 786.063048] LustreError: 24236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790391747 with bad export cookie 14216670964064611605 [ 786.064991] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 786.088671] LustreError: 24236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 786.538423] Lustre: server umount lustre-MDT0001 complete [ 798.022615] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 801.167863] Key type lgssc unregistered [ 801.525851] LNet: 28244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.531912] LNetError: 28244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 801.560561] LNet: Removed LNI 192.168.201.119@tcp [ 802.450824] Key type .llcrypt unregistered [ 802.454719] Key type ._llcrypt unregistered [ 828.938268] Key type ._llcrypt registered [ 828.941471] Key type .llcrypt registered [ 829.085289] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 830.588938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 830.720276] alg: No test for adler32 (adler32-zlib) [ 831.861742] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 832.125711] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 833.843087] Key type lgssc registered [ 834.925713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 843.510407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 843.525449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.031584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.930677] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 855.523531] LustreError: 29661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 855.548907] LustreError: 29661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 860.250046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.704654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 865.664537] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 873.854557] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 880.515734] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 895.920974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 896.484035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 904.677231] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 913.655761] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 914.937513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 925.152038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 925.166523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.176932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.184346] Lustre: Skipped 1 previous similar message [ 925.209668] Lustre: Skipped 1 previous similar message [ 927.638535] Lustre: server umount lustre-OST0000 complete [ 932.324911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 932.336056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.350253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.394160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.394425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.538912] Lustre: server umount lustre-MDT0000 complete [ 940.512643] LustreError: 29661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 942.556773] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790391903 with bad export cookie 3227837534956705320 [ 942.570202] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 942.581410] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 943.060301] Lustre: server umount lustre-MDT0001 complete [ 956.113696] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 959.381889] Key type lgssc unregistered [ 959.746791] LNet: 33540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.755475] LNetError: 33540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.788135] LNet: Removed LNI 192.168.201.119@tcp [ 960.733203] Key type .llcrypt unregistered [ 960.736261] Key type ._llcrypt unregistered [ 983.702825] Key type ._llcrypt registered [ 983.704981] Key type .llcrypt registered [ 983.796870] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 985.156936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 985.284945] alg: No test for adler32 (adler32-zlib) [ 986.325824] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 986.519967] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 988.192730] Key type lgssc registered [ 989.917883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 999.213561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 999.258738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.167197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.686574] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1011.681750] LustreError: 34950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1011.699567] LustreError: 34950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1015.204862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.468513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1019.591577] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1028.065726] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1033.912393] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1041.853462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1042.383960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.534648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1049.399737] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1057.613422] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1070.053234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1070.056660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1070.064603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1070.075794] Lustre: Skipped 1 previous similar message [ 1070.095614] Lustre: Skipped 1 previous similar message [ 1073.856945] Lustre: server umount lustre-OST0000 complete [ 1080.289075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.292595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1080.309069] Lustre: Skipped 1 previous similar message [ 1084.646420] Lustre: server umount lustre-MDT0000 complete [ 1085.420484] LustreError: 34949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.452620] LustreError: 34949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1088.813390] LustreError: 34932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790392050 with bad export cookie 581055858688872768 [ 1088.816943] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1088.828163] LustreError: 34932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1089.174816] Lustre: server umount lustre-MDT0001 complete [ 1099.785337] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1102.757235] Key type lgssc unregistered [ 1103.089255] LNet: 38487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.111972] LNetError: 38487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1103.120989] LNet: Removed LNI 192.168.201.119@tcp [ 1103.952334] Key type .llcrypt unregistered [ 1103.954958] Key type ._llcrypt unregistered [ 1126.321321] Key type ._llcrypt registered [ 1126.322827] Key type .llcrypt registered [ 1126.454209] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1127.558040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1127.585880] alg: No test for adler32 (adler32-zlib) [ 1128.649490] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1128.882974] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1130.543663] Key type lgssc registered [ 1131.720660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1140.359532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1140.391197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.815511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.702302] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1151.970253] LustreError: 39903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.986484] LustreError: 39903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1155.117342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1155.553127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1160.763414] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1167.883202] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1173.002423] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1179.262047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1179.642150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.071578] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1186.827823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1193.327585] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1200.129356] Lustre: lustre-OST0000: Not available for connect from 192.168.201.19@tcp (stopping) [ 1201.120457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1201.124583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.136393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1202.148055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1202.150214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.150996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1205.300199] Lustre: server umount lustre-OST0000 complete [ 1227.242533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1227.247899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.266230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1227.278439] Lustre: Skipped 1 previous similar message [ 1230.912153] Lustre: server umount lustre-MDT0000 complete [ 1233.888266] LustreError: 39900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.897230] LustreError: 39900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1234.373396] LustreError: 41745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790392195 with bad export cookie 17873069416665290414 [ 1234.386144] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1234.386534] LustreError: 41745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1240.848177] Lustre: server umount lustre-MDT0001 complete [ 1251.357309] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1254.379593] Key type lgssc unregistered [ 1254.719557] LNet: 43610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1254.730795] LNetError: 43610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.784261] LNet: Removed LNI 192.168.201.119@tcp [ 1256.634529] Key type .llcrypt unregistered [ 1256.637551] Key type ._llcrypt unregistered [ 1277.377872] Key type ._llcrypt registered [ 1277.379495] Key type .llcrypt registered [ 1277.475867] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1278.439339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1278.564425] alg: No test for adler32 (adler32-zlib) [ 1279.612284] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1279.859319] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1281.519165] Key type lgssc registered [ 1282.761716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1290.382660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1290.408566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.854412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1296.324671] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1301.984595] LustreError: 45014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.995111] LustreError: 45014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1303.541666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.768492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1307.523848] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1312.937555] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1317.043123] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1321.552914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1321.914229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.573827] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1331.955270] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1337.282143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1342.435649] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1342.441087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1342.454606] Lustre: Skipped 1 previous similar message [ 1342.464712] Lustre: Skipped 2 previous similar messages [ 1347.066406] Lustre: server umount lustre-MDT0000 complete [ 1347.556324] LustreError: 46852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.583179] LustreError: 46852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1350.201297] LustreError: 44995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790392311 with bad export cookie 15750455618384720761 [ 1350.206748] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1350.208743] LustreError: 44995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1352.673083] LustreError: 46852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.673683] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.698513] LustreError: 46852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1352.711794] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.19@tcp (stopping) [ 1352.725136] Lustre: Skipped 1 previous similar message [ 1352.757572] Lustre: Skipped 1 previous similar message [ 1355.680329] LustreError: 46852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.681429] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1356.544155] Lustre: server umount lustre-MDT0001 complete [ 1377.085183] Lustre: server umount lustre-OST0000 complete [ 1389.805893] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1391.865327] Key type lgssc unregistered [ 1392.090888] LNet: 48879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.094895] LNetError: 48879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.109181] LNet: Removed LNI 192.168.201.119@tcp [ 1392.781758] Key type .llcrypt unregistered [ 1392.783379] Key type ._llcrypt unregistered [ 1411.356475] Key type ._llcrypt registered [ 1411.358738] Key type .llcrypt registered [ 1411.454373] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1412.295401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1412.319506] alg: No test for adler32 (adler32-zlib) [ 1413.320163] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1413.571805] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1415.247467] Key type lgssc registered [ 1416.111874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1422.164127] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1422.182272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1437.727454] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1442.847363] LustreError: 50241:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 1469.535139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.479366] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1480.322245] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1644.609375] Lustre: server umount lustre-OST0000 complete [ 1657.080364] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1659.105942] Key type lgssc unregistered [ 1659.298595] LNet: 51914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.301543] LNetError: 51914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.325797] LNet: Removed LNI 192.168.201.119@tcp [ 1661.047800] Key type .llcrypt unregistered [ 1661.053178] Key type ._llcrypt unregistered [ 1676.739023] Key type ._llcrypt registered [ 1676.740739] Key type .llcrypt registered [ 1676.786574] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1677.225063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1677.261422] alg: No test for adler32 (adler32-zlib) [ 1678.162068] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1678.275801] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1679.872969] Key type lgssc registered [ 1680.402737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1684.137742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1684.145427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.393921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1687.557628] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1692.606194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.861252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1695.153173] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1698.667179] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1701.643181] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1705.573631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1705.760467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.438675] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1709.863913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1712.245631] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1715.169325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.182484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1718.243762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1718.253641] Lustre: Skipped 1 previous similar message [ 1719.441048] Lustre: server umount lustre-OST0000 complete [ 1723.361163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.362399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1723.367187] Lustre: Skipped 1 previous similar message [ 1727.435916] Lustre: server umount lustre-MDT0000 complete [ 1728.481837] LustreError: 54031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1728.492970] LustreError: 54031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1729.395552] LustreError: 53283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790392690 with bad export cookie 10431199450230337220 [ 1729.396919] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1729.401511] LustreError: 53283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1729.539021] Lustre: server umount lustre-MDT0001 complete [ 1734.870472] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1736.439382] Key type lgssc unregistered [ 1736.639791] LNet: 56828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.667615] LNetError: 56828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.705825] LNet: Removed LNI 192.168.201.119@tcp [ 1738.218132] Key type .llcrypt unregistered [ 1738.219797] Key type ._llcrypt unregistered [ 1750.720424] Key type ._llcrypt registered [ 1750.721940] Key type .llcrypt registered [ 1750.777947] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1751.223272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1751.255411] alg: No test for adler32 (adler32-zlib) [ 1752.170992] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1752.295896] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1753.887815] Key type lgssc registered [ 1754.407227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1758.123926] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1758.143083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1773.599411] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1778.719339] LustreError: 58188:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 1805.343275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.751365] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1812.967462] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1816.466726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.756606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.807451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1818.987592] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1823.359237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1823.610048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1825.627021] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1829.823944] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1833.144598] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1837.536106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1837.540178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1837.546762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1839.074640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1839.077617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.084569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1840.373594] Lustre: server umount lustre-OST0000 complete [ 1856.255131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.262544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1856.268435] Lustre: Skipped 2 previous similar messages [ 1859.363606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1859.365948] Lustre: Skipped 1 previous similar message [ 1862.437716] Lustre: server umount lustre-MDT0000 complete [ 1864.538376] LustreError: 58194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790392825 with bad export cookie 8916574040255042481 [ 1864.540876] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1864.544925] LustreError: 58194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1870.776773] Lustre: server umount lustre-MDT0001 complete [ 1876.025753] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1877.771853] Key type lgssc unregistered [ 1877.999760] LNet: 61926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.004262] LNetError: 61926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1878.016245] LNet: Removed LNI 192.168.201.119@tcp [ 1878.478647] Key type .llcrypt unregistered [ 1878.480133] Key type ._llcrypt unregistered [ 1890.540864] Key type ._llcrypt registered [ 1890.542571] Key type .llcrypt registered [ 1890.589872] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1891.191482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1891.203066] alg: No test for adler32 (adler32-zlib) [ 1892.164471] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1892.373542] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1894.007472] Key type lgssc registered [ 1894.799181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1899.406540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1899.416445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.664066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.846842] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1907.352534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.581649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1909.748722] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1912.845627] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1915.808891] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1919.202059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1919.373034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.380750] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1925.542788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1926.422699] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1940.960924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1940.966104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.971682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1943.009154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1943.009299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1943.015286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.969517] Lustre: server umount lustre-OST0000 complete [ 1948.129809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.131255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1948.135307] Lustre: Skipped 1 previous similar message [ 1948.140352] Lustre: Skipped 2 previous similar messages [ 1953.249466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.252744] Lustre: Skipped 1 previous similar message [ 1953.922864] Lustre: server umount lustre-MDT0000 complete [ 1955.842959] LustreError: 63293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790392917 with bad export cookie 8821581359710002978 [ 1955.848156] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1955.850753] LustreError: 63293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1956.040935] Lustre: server umount lustre-MDT0001 complete [ 1960.822347] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1962.300294] Key type lgssc unregistered [ 1962.437730] LNet: 66863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.440867] LNetError: 66863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.454762] LNet: Removed LNI 192.168.201.119@tcp [ 1962.865772] Key type .llcrypt unregistered [ 1962.867299] Key type ._llcrypt unregistered [ 1976.140661] Key type ._llcrypt registered [ 1976.142453] Key type .llcrypt registered [ 1976.202075] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1976.625544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1976.693506] alg: No test for adler32 (adler32-zlib) [ 1977.618240] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1977.756293] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1979.359172] Key type lgssc registered [ 1980.001119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1984.830162] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 23:22:25 (1790392945) [ 1989.059278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1989.069777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.296300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.418351] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1996.498322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.695430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1998.410260] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2001.468586] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2003.759095] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2006.841909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2007.057775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.687893] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2012.977103] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2015.270827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2026.542512] Lustre: Failing over lustre-MDT0000 [ 2026.661239] Lustre: server umount lustre-MDT0000 complete [ 2027.488940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2027.494788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.504812] LustreError: 68540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.521040] LustreError: 68540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2030.575445] LustreError: 68535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.914992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.977758] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2031.040299] Lustre: *** cfs_fail_loc=135, val=0*** [ 2031.088383] LustreError: 71646:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2031.093289] LustreError: 71646:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.119@tcp: cfg command failed: rc = -2 [ 2031.097380] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2031.097380] [ 2031.101172] LustreError: MGC192.168.201.119@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2031.106773] LustreError: 71634:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2031.110226] LustreError: 71634:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2031.114487] LustreError: 71634:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2031.116951] LustreError: 71634:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2031.129892] Lustre: server umount lustre-MDT0000 complete [ 2031.131822] LustreError: 71634:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2031.584114] LustreError: 68539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2031.585659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.591112] LustreError: 68539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2031.596309] Lustre: Skipped 1 previous similar message [ 2034.081428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.149814] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2034.270666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.298281] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2035.684203] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2036.172810] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2039.275697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2039.285193] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2039.304681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2041.823991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2041.827500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.834558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2044.387713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2044.388868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.388901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2046.690232] Lustre: server umount lustre-OST0000 complete [ 2049.505511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2049.505698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.515179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.519499] Lustre: Skipped 2 previous similar messages [ 2054.422297] Lustre: server umount lustre-MDT0000 complete [ 2054.625066] LustreError: 68539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.632241] LustreError: 68539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2055.900093] LustreError: 70381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393017 with bad export cookie 6622875268843315782 [ 2055.903265] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2055.904133] LustreError: 70381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2056.053634] Lustre: server umount lustre-MDT0001 complete [ 2060.413298] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2061.644388] Key type lgssc unregistered [ 2061.791385] LNet: 73414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.795126] LNetError: 73414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.806363] LNet: Removed LNI 192.168.201.119@tcp [ 2062.203855] Key type .llcrypt unregistered [ 2062.205545] Key type ._llcrypt unregistered [ 2073.280273] Key type ._llcrypt registered [ 2073.281843] Key type .llcrypt registered [ 2073.326974] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2073.765703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2073.777050] alg: No test for adler32 (adler32-zlib) [ 2074.692715] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2074.813100] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2076.415195] Key type lgssc registered [ 2076.925258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2080.384328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2080.394938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.616780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.334677] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2087.049545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.229645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2088.997512] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2092.066824] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2094.367184] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2097.291992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2097.483975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.084871] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2103.408802] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2105.832085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2111.266744] Lustre: Failing over lustre-MDT0000 [ 2111.431295] Lustre: server umount lustre-MDT0000 complete [ 2112.994957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2112.999748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.007339] LustreError: 74801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.014697] LustreError: 74801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2114.030039] LustreError: 74797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2116.065280] LustreError: 74800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2116.066795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.072912] LustreError: 74800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2116.077526] Lustre: Skipped 1 previous similar message [ 2116.116962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.179280] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2116.241769] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2116.304700] LustreError: 77885:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2116.312900] LustreError: 77885:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.119@tcp: cfg command failed: rc = -9 [ 2116.320740] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2116.320740] [ 2116.325768] LustreError: MGC192.168.201.119@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2116.332605] LustreError: 77873:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2116.335587] LustreError: 77873:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2116.338698] LustreError: 77873:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2116.341694] LustreError: 77873:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2116.355483] Lustre: server umount lustre-MDT0000 complete [ 2116.356750] LustreError: 77873:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2119.141037] LustreError: 76536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.643431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2119.710769] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2119.859715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.901457] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2122.103429] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2124.268367] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2125.293474] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2125.316070] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2125.344815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2128.352079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2128.358926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2128.367216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2130.401167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2130.405871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2130.409302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.218751] Lustre: server umount lustre-OST0000 complete [ 2135.520397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2135.523815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.526700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2135.532548] Lustre: Skipped 1 previous similar message [ 2135.540106] Lustre: Skipped 2 previous similar messages [ 2140.642285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2140.645670] Lustre: Skipped 1 previous similar message [ 2141.478916] Lustre: server umount lustre-MDT0000 complete [ 2143.553699] LustreError: 76641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393104 with bad export cookie 10692662664105909301 [ 2143.557577] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2143.559409] LustreError: 76641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2143.727343] Lustre: server umount lustre-MDT0001 complete [ 2149.415129] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2150.932939] Key type lgssc unregistered [ 2151.128744] LNet: 79661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.132356] LNetError: 79661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2152.168468] LNet: Removed LNI 192.168.201.119@tcp [ 2152.649260] Key type .llcrypt unregistered [ 2152.650572] Key type ._llcrypt unregistered [ 2159.143451] Key type ._llcrypt registered [ 2159.144613] Key type .llcrypt registered [ 2159.205137] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2159.336478] Key type .llcrypt unregistered [ 2159.338478] Key type ._llcrypt unregistered [ 2171.510450] Key type ._llcrypt registered [ 2171.511940] Key type .llcrypt registered [ 2171.567309] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2172.262402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2172.284316] alg: No test for adler32 (adler32-zlib) [ 2173.199032] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2173.329657] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2174.927249] Key type lgssc registered [ 2175.498044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2179.315459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2179.324248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.565600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.068239] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2188.469789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.668453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2190.876047] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2195.085664] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2198.534787] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2202.660151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2202.839409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.914552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2206.393768] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2211.366203] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2220.512706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2220.516085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2220.521346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2223.584708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2223.589482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2223.594152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.685872] Lustre: server umount lustre-OST0000 complete [ 2229.735377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2229.744895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.753139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2229.755940] Lustre: Skipped 1 previous similar message [ 2233.826943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.827164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.944148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2238.948413] Lustre: Skipped 2 previous similar messages [ 2243.039132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2243.118601] Lustre: server umount lustre-MDT0000 complete [ 2244.657204] LustreError: 81512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393205 with bad export cookie 9924814570119639716 [ 2244.661803] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2244.662814] LustreError: 81512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2244.813072] Lustre: server umount lustre-MDT0001 complete [ 2248.878778] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2250.092737] Key type lgssc unregistered [ 2250.240578] LNet: 85061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.247126] LNetError: 85061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2250.257445] LNet: Removed LNI 192.168.201.119@tcp [ 2250.618395] Key type .llcrypt unregistered [ 2250.619703] Key type ._llcrypt unregistered [ 2261.646818] Key type ._llcrypt registered [ 2261.648512] Key type .llcrypt registered [ 2261.697819] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2262.080702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2262.092091] alg: No test for adler32 (adler32-zlib) [ 2263.019929] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2263.137956] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2264.767188] Key type lgssc registered [ 2265.316576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2268.928497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2268.940884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2270.125568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.882924] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2275.817554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.004797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2277.949150] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2281.272492] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2283.578955] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2286.487474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2286.687088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.384907] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2292.817918] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2294.889966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2306.529587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2306.530686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.534299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.539539] Lustre: Skipped 1 previous similar message [ 2306.545129] Lustre: Skipped 1 previous similar message [ 2307.648933] Lustre: server umount lustre-OST0000 complete [ 2311.535900] Lustre: server umount lustre-MDT0000 complete [ 2311.651030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.652140] LustreError: 88934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.657721] Lustre: Skipped 1 previous similar message [ 2311.665190] LustreError: 88934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2313.498253] LustreError: 86426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393274 with bad export cookie 1681879261916011935 [ 2313.502459] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2313.505720] LustreError: 86426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2313.671569] Lustre: server umount lustre-MDT0001 complete [ 2318.120430] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2319.453872] Key type lgssc unregistered [ 2319.601986] LNet: 89993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2319.608662] LNetError: 89993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2319.627096] LNet: Removed LNI 192.168.201.119@tcp [ 2320.013777] Key type .llcrypt unregistered [ 2320.015626] Key type ._llcrypt unregistered [ 2331.174921] Key type ._llcrypt registered [ 2331.176076] Key type .llcrypt registered [ 2331.224660] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2331.864471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2331.877381] alg: No test for adler32 (adler32-zlib) [ 2332.766177] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2332.898531] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2334.519160] Key type lgssc registered [ 2335.083568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2338.849268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2338.866798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.083379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.828129] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2345.681387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.847149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2347.641179] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2351.149773] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2354.011727] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2357.765117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2357.975446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.957563] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2364.812881] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2365.158607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2375.648713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2375.652688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.658568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2376.674224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2376.674468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.675452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2381.030139] Lustre: server umount lustre-OST0000 complete [ 2386.913265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.914384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2386.914982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2386.918962] Lustre: Skipped 1 previous similar message [ 2386.925271] Lustre: Skipped 2 previous similar messages [ 2388.800487] Lustre: server umount lustre-MDT0000 complete [ 2390.630464] LustreError: 91360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393351 with bad export cookie 13873324641481932285 [ 2390.634992] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2390.635941] LustreError: 91360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2390.795295] Lustre: server umount lustre-MDT0001 complete [ 2395.224831] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2396.499882] Key type lgssc unregistered [ 2396.650528] LNet: 94910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.653340] LNetError: 94910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2396.666175] LNet: Removed LNI 192.168.201.119@tcp [ 2397.093816] Key type .llcrypt unregistered [ 2397.094988] Key type ._llcrypt unregistered [ 2407.502331] Key type ._llcrypt registered [ 2407.503656] Key type .llcrypt registered [ 2407.549733] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2407.984324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2408.005702] alg: No test for adler32 (adler32-zlib) [ 2408.874614] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2408.968219] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2410.559162] Key type lgssc registered [ 2410.989398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2414.149943] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2414.156470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2429.599459] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2434.719228] LustreError: 96271:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 2461.343231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.906204] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2467.147601] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2469.288220] Lustre: server umount lustre-OST0000 complete [ 2472.654180] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 23:30:33 (1790393433) [ 2475.068590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.223805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2476.698855] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2479.983062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.146860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2481.674392] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2484.296345] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2486.263951] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2488.666644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2488.792464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.026544] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2493.732169] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2501.094649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2510.305266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2510.307507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.311844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2511.328567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2511.328913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.330179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2513.809496] Lustre: server umount lustre-OST0000 complete [ 2515.935838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2515.938370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.943411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2515.946444] Lustre: Skipped 1 previous similar message [ 2521.376611] Lustre: server umount lustre-MDT0000 complete [ 2521.568378] LustreError: 98633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.578510] LustreError: 98633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2522.885616] LustreError: 99747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393484 with bad export cookie 10416245122021762546 [ 2522.889499] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2522.890570] LustreError: 99747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2523.016191] Lustre: server umount lustre-MDT0001 complete [ 2527.081390] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2528.255154] Key type lgssc unregistered [ 2528.387500] LNet: 101608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2528.390755] LNetError: 101608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2528.400291] LNet: Removed LNI 192.168.201.119@tcp [ 2528.718775] Key type .llcrypt unregistered [ 2528.720180] Key type ._llcrypt unregistered [ 2544.719869] Key type ._llcrypt registered [ 2544.721240] Key type .llcrypt registered [ 2544.765423] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2545.143265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.172292] alg: No test for adler32 (adler32-zlib) [ 2546.052799] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2546.164334] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2547.759145] Key type lgssc registered [ 2548.138515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2551.181344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2551.189620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.379687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.882306] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2557.028482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.197584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2558.815924] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2561.972074] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2564.196825] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2567.100334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2567.231378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.473730] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2572.246206] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2577.509170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2583.008616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2583.011483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2583.012393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2583.014023] Lustre: Skipped 1 previous similar message [ 2586.385835] Lustre: server umount lustre-OST0000 complete [ 2588.128278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2588.129042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.134636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2588.136294] Lustre: Skipped 1 previous similar message [ 2588.138723] Lustre: Skipped 1 previous similar message [ 2593.248113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2593.250306] Lustre: Skipped 1 previous similar message [ 2593.816891] Lustre: server umount lustre-MDT0000 complete [ 2595.104247] LustreError: 104497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393556 with bad export cookie 5690568120236421550 [ 2595.107611] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2595.111791] LustreError: 104497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2595.234279] Lustre: server umount lustre-MDT0001 complete [ 2599.061699] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2600.193260] Key type lgssc unregistered [ 2600.331657] LNet: 107294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2600.334554] LNetError: 107294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2600.343303] LNet: Removed LNI 192.168.201.119@tcp [ 2600.664945] Key type .llcrypt unregistered [ 2600.665889] Key type ._llcrypt unregistered [ 2608.337637] Key type ._llcrypt registered [ 2608.338958] Key type .llcrypt registered [ 2608.376995] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2608.723123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2608.752199] alg: No test for adler32 (adler32-zlib) [ 2609.610022] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2609.702447] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2611.279169] Key type lgssc registered [ 2611.704307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2614.665036] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2614.676563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2630.111344] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2635.231293] LustreError: 108510:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 2661.855211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.732229] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2666.209734] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2668.548628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.664390] LustreError: MGC192.168.201.119@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2668.669182] LustreError: 109534:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2668.671409] LustreError: 109534:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2668.674268] LustreError: 109534:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2668.707708] Lustre: server umount lustre-MDT0000 complete [ 2668.709814] LustreError: 109534:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2680.834059] Lustre: server umount lustre-OST0000 complete [ 2684.186076] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 2687.636972] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2699.293058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.372262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2699.382111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2699.418584] Lustre: lustre-MDT0000: new disk, initializing [ 2699.444170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.450478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2700.751879] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2704.797952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.841451] Lustre: 112320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2704.866799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2704.869343] Lustre: Skipped 1 previous similar message [ 2704.902447] Lustre: lustre-MDT0001: new disk, initializing [ 2704.941649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2704.959835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2704.963991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2706.299626] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2709.998546] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2711.814869] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2714.078192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2714.186273] Lustre: lustre-OST0000: new disk, initializing [ 2714.188201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2714.192327] Lustre: 114288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2714.225027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.871763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2714.875858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2714.922702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2716.201982] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2719.849802] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2721.205801] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2721.280596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2722.586081] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2722.657624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2725.345539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2725.346812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.348359] LustreError: Skipped 1 previous similar message [ 2725.348950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2725.352475] Lustre: Skipped 1 previous similar message [ 2725.356262] Lustre: Skipped 1 previous similar message [ 2729.357294] Lustre: server umount lustre-OST0000 complete [ 2735.584483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2735.584792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.591564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.594592] Lustre: Skipped 1 previous similar message [ 2736.728933] Lustre: server umount lustre-MDT0000 complete [ 2737.873482] LustreError: 112312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393699 with bad export cookie 17297529354303359648 [ 2737.876718] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2737.877416] LustreError: 112312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2737.999323] Lustre: server umount lustre-MDT0001 complete [ 2740.818073] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 23:35:02 (1790393702) [ 2741.953662] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2747.104267] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 2750.051938] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2761.432648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.512066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2761.522497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2761.558925] Lustre: lustre-MDT0000: new disk, initializing [ 2761.578583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.585143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2762.959955] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2767.057647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.102791] Lustre: 119179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2767.135168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2767.136273] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2767.139090] Lustre: Skipped 1 previous similar message [ 2767.177280] Lustre: lustre-MDT0001: new disk, initializing [ 2767.217597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2767.221362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2768.536591] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2772.232570] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2774.004765] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2776.187781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2776.284985] Lustre: lustre-OST0000: new disk, initializing [ 2776.287551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2776.288326] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2776.292213] Lustre: 121145:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2776.326250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.328870] Lustre: Skipped 1 previous similar message [ 2778.101430] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2781.673131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2781.678934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2781.685625] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2781.691989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2783.074858] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2783.149187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2784.458476] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2784.534099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2786.784682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2786.786058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.788238] LustreError: Skipped 1 previous similar message [ 2786.793169] Lustre: Skipped 1 previous similar message [ 2786.794134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2786.799757] Lustre: Skipped 1 previous similar message [ 2787.216653] Lustre: server umount lustre-OST0000 complete [ 2791.904601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.909883] Lustre: Skipped 2 previous similar messages [ 2794.520940] Lustre: server umount lustre-MDT0000 complete [ 2795.707948] LustreError: 119171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393756 with bad export cookie 17297529354303360516 [ 2795.710447] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2795.711351] LustreError: 119171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2795.822070] Lustre: server umount lustre-MDT0001 complete [ 2798.456323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.622439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2799.861826] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2802.445086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.859479] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2806.065537] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2807.726721] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2809.866056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.826487] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2814.214689] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2827.054424] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2829.793976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2829.794367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.795390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2829.795393] Lustre: Skipped 2 previous similar messages [ 2829.802333] Lustre: Skipped 1 previous similar message [ 2830.028420] Lustre: server umount lustre-OST0000 complete [ 2833.296687] Lustre: server umount lustre-MDT0000 complete [ 2833.376084] LustreError: 123377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.380582] LustreError: 123377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2834.454258] LustreError: 123340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393795 with bad export cookie 17297529354303361398 [ 2834.457873] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2834.458021] LustreError: 123340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2834.562560] Lustre: server umount lustre-MDT0001 complete [ 2837.692873] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2838.623498] Key type lgssc unregistered [ 2838.737336] LNet: 127003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2838.739498] LNetError: 127003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2838.745291] LNet: Removed LNI 192.168.201.119@tcp [ 2839.016898] Key type .llcrypt unregistered [ 2839.018124] Key type ._llcrypt unregistered [ 2844.347543] Key type ._llcrypt registered [ 2844.348578] Key type .llcrypt registered [ 2844.382418] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 2850.000201] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2850.273556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2850.281404] alg: No test for adler32 (adler32-zlib) [ 2851.120444] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2851.202521] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2852.775121] Key type lgssc registered [ 2853.137291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2863.978884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2863.984511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.058406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2865.067817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2865.106706] Lustre: lustre-MDT0000: new disk, initializing [ 2865.129691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.135649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2866.430015] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2870.341087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2870.377581] Lustre: 130121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2870.397391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2870.398865] Lustre: Skipped 1 previous similar message [ 2870.430419] Lustre: lustre-MDT0001: new disk, initializing [ 2870.460446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2870.471568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2870.474825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2871.725852] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2875.260958] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2876.936507] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2878.967557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2879.071881] Lustre: lustre-OST0000: new disk, initializing [ 2879.073352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2879.075268] Lustre: 132088:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2879.101837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2880.565651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2880.568626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2880.596647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2880.795905] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2884.511796] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2886.015502] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2886.091367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2887.485621] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2887.569619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2890.725627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2890.725825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2890.728477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2890.731281] Lustre: Skipped 1 previous similar message [ 2894.358286] Lustre: server umount lustre-OST0000 complete [ 2895.840613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.841652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.845216] Lustre: Skipped 1 previous similar message [ 2895.846944] Lustre: Skipped 1 previous similar message [ 2900.960420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2900.963149] Lustre: Skipped 2 previous similar messages [ 2901.781189] Lustre: server umount lustre-MDT0000 complete [ 2902.895056] LustreError: 130116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393864 with bad export cookie 6870307991065141432 [ 2902.897566] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2902.898640] LustreError: 130116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2902.999541] Lustre: server umount lustre-MDT0001 complete [ 2905.432176] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 23:37:46 (1790393866) [ 2907.460857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.612309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.809753] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2911.338553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.673965] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2914.790458] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2916.409302] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2921.120879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.125529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.127482] Lustre: Skipped 1 previous similar message [ 2923.157880] Lustre: server umount lustre-MDT0000 complete [ 2924.408466] LustreError: 134476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393885 with bad export cookie 6870307991065142300 [ 2924.411523] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2924.414108] LustreError: 134476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2924.528145] Lustre: server umount lustre-MDT0001 complete [ 2927.265476] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 23:38:08 (1790393888) [ 2929.430486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2944.863651] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2949.983466] LustreError: 136908:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 2975.583413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.586996] Lustre: Skipped 1 previous similar message [ 2977.538423] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2979.908834] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2980.609272] Lustre: server umount lustre-OST0000 complete [ 2983.279924] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 23:39:04 (1790393944) [ 2985.483925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.643415] LustreError: 138352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.650670] LustreError: 138352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2985.670034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.068032] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2989.938367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2991.278024] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2993.458787] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2995.105470] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2997.159925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2999.016318] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3001.129608] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3015.650411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3015.650436] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3015.651959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.653875] LustreError: Skipped 1 previous similar message [ 3015.657964] Lustre: Skipped 2 previous similar messages [ 3019.162617] Lustre: server umount lustre-MDT0000 complete [ 3020.442308] LustreError: 138335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393981 with bad export cookie 6870307991065142937 [ 3020.445911] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3020.448717] LustreError: 138335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3020.564534] Lustre: server umount lustre-MDT0001 complete [ 3032.086838] Lustre: server umount lustre-OST0000 complete [ 3034.682636] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 23:39:56 (1790393996) [ 3036.890585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3037.069758] LustreError: 142022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.097990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3037.100832] Lustre: Skipped 2 previous similar messages [ 3038.459222] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3041.308290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.837301] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3045.165938] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3046.977800] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3049.132668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3051.074924] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3053.435875] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3054.838328] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3063.591058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3064.081981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3065.540331] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3065.618430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3066.847789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3066.850225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3066.855622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3066.857969] Lustre: Skipped 2 previous similar messages [ 3068.899257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3072.395891] Lustre: server umount lustre-OST0000 complete [ 3079.772239] Lustre: server umount lustre-MDT0000 complete [ 3081.028333] LustreError: 142004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394042 with bad export cookie 6870307991065143994 [ 3081.030923] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3081.032618] LustreError: 142004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3081.151932] Lustre: server umount lustre-MDT0001 complete [ 3083.800380] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 23:40:45 (1790394045) [ 3085.909052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3101.343458] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3106.463388] LustreError: 146239:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 3132.063278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.066030] Lustre: Skipped 2 previous similar messages [ 3133.910679] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3136.233239] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3138.443392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.579601] LustreError: 146264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3138.622241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3139.884538] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3142.589861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3143.959409] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3146.138211] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3147.818517] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3149.098802] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3149.161542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3150.444210] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3150.501056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3152.354688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.355792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3152.357705] Lustre: Skipped 4 previous similar messages [ 3152.360183] Lustre: Skipped 6 previous similar messages [ 3157.264705] Lustre: server umount lustre-MDT0000 complete [ 3157.473366] LustreError: 147274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3157.477352] LustreError: 147274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3158.497192] LustreError: 146245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394119 with bad export cookie 6870307991065145002 [ 3158.500118] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3158.500620] LustreError: 146245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3158.595393] Lustre: server umount lustre-MDT0001 complete [ 3168.918288] Lustre: server umount lustre-OST0000 complete [ 3171.529254] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 23:42:12 (1790394132) [ 3173.658311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3189.087709] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3194.207418] LustreError: 150427:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 3219.807313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.809988] Lustre: Skipped 2 previous similar messages [ 3221.531059] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3223.689206] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3225.632768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3225.754349] LustreError: 150452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.800202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3226.979552] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3229.502533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3230.948487] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3233.190432] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3234.886312] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3236.983886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3237.042416] Lustre: lustre-OST0001: new disk, initializing [ 3237.043922] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3237.046854] Lustre: 153293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3238.787042] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3242.254483] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3242.485723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3242.488410] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3242.516201] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3243.517206] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3243.583925] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3244.888063] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3244.959029] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3246.559932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3246.562423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.565688] Lustre: Skipped 1 previous similar message [ 3246.567190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3251.705948] Lustre: server umount lustre-OST0000 complete [ 3252.706563] LustreError: 150443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.713455] LustreError: 150443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3257.825755] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3258.953633] Lustre: server umount lustre-OST0001 complete [ 3266.200384] Lustre: server umount lustre-MDT0000 complete [ 3267.347960] LustreError: 152199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394228 with bad export cookie 6870307991065145954 [ 3267.349913] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3267.352878] LustreError: 152199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3267.457530] Lustre: server umount lustre-MDT0001 complete [ 3274.485805] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 23:43:55 (1790394235) [ 3274.990104] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3275.458374] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3275.935889] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 23:43:57 (1790394237) [ 3277.997464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3278.085538] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3278.094114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3279.427566] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3282.131069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.167530] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3282.177862] Lustre: 156848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3283.568616] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3285.864714] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3287.548945] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3289.602306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3289.665831] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3291.541371] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3293.690401] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3294.216707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3294.969250] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3295.034431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3296.260811] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3296.321789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3299.298469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3299.298535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.302151] LustreError: Skipped 3 previous similar messages [ 3299.306989] Lustre: Skipped 6 previous similar messages [ 3302.990275] Lustre: server umount lustre-OST0000 complete [ 3316.521414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3318.510254] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3319.716344] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3319.720066] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3319.720102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3320.664348] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3322.024318] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3322.086722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3323.240873] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3323.300579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3324.870445] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93c209647800.ost_server_uuid 50 [ 3325.321362] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93c209647800.ost_server_uuid in FULL state after 0 sec [ 3325.372640] LustreError: 156856:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.19@tcp) returned error from blocking AST (req@0000000022eaa891 x1877363981994752 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b1225eea/0x5f583cb7a8f73bae lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.19@tcp remote: 0x70beab13de93d07e expref: 5 pid: 156857 timeout: 3425 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3325.382160] LustreError: lustre-MDT0000: A client on nid 192.168.201.19@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3325.384648] LustreError: 156845:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.19@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b1225eea/0x5f583cb7a8f73bae lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.19@tcp remote: 0x70beab13de93d07e expref: 6 pid: 156857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3330.018515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3330.021334] Lustre: Skipped 10 previous similar messages [ 3332.427518] Lustre: server umount lustre-OST0000 complete [ 3339.803463] Lustre: server umount lustre-MDT0000 complete [ 3340.256240] LustreError: 156860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.263436] LustreError: 156860:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3341.047302] LustreError: 156841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394302 with bad export cookie 6870307991065147137 [ 3341.049726] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3341.051261] LustreError: 156841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3347.245035] Lustre: server umount lustre-MDT0001 complete [ 3350.803308] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3351.857594] Key type lgssc unregistered [ 3351.989332] LNet: 162952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3351.991848] LNetError: 162952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3352.005357] LNet: Removed LNI 192.168.201.119@tcp [ 3352.333838] Key type .llcrypt unregistered [ 3352.335656] Key type ._llcrypt unregistered [ 3360.679341] Key type ._llcrypt registered [ 3360.680356] Key type .llcrypt registered [ 3360.715329] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3361.079615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3361.089201] alg: No test for adler32 (adler32-zlib) [ 3361.939025] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 3362.024164] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3363.607126] Key type lgssc registered [ 3363.927778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3366.449994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3366.455944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.580538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.832018] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3371.507371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.609202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3372.821363] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3375.105719] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3376.792056] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3378.813495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3378.917421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.683540] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3382.939650] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3392.229900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3395.611088] Lustre: Failing over lustre-MDT0000 [ 3395.777659] Lustre: server umount lustre-MDT0000 complete [ 3397.087983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3397.090567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.094830] LustreError: 164356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3397.101228] LustreError: 164356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3397.600646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.601741] LustreError: 164334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3397.604606] Lustre: Skipped 1 previous similar message [ 3402.720486] LustreError: 164338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.725113] LustreError: 164338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3406.696099] LustreError: 164317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.19@tcp arrived at 1790394367 with bad export cookie 10093173268016880433 [ 3406.699885] LustreError: 164317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3407.841739] LustreError: 164337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.848246] LustreError: 164337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3409.084239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3409.130497] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3409.225356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.258118] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3410.614517] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3411.384247] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3411.387387] Lustre: lustre-MDT0000: Denying connection for new client 627d0b45-d107-4bf3-b473-006f14ee37cb (at 192.168.201.19@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 3414.501473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3419.617531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.619037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3419.621895] Lustre: Skipped 1 previous similar message [ 3424.208166] Lustre: server umount lustre-OST0000 complete [ 3425.350794] LustreError: 168081:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3425.353216] Lustre: 167431:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3425.355549] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3425.361314] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3425.366534] LustreError: 167431:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000aed6bc9f x1877364517671296/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3425.373707] LustreError: 167431:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3425.376015] LustreError: 167431:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3425.378280] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 3425.466873] Lustre: server umount lustre-MDT0000 complete [ 3426.563159] LustreError: 165071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394387 with bad export cookie 10093173268016880755 [ 3426.566457] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3426.567501] LustreError: 163647:0:(import.c:751:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3426.567939] LustreError: 165071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3432.749930] Lustre: server umount lustre-MDT0001 complete [ 3436.441921] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3437.527805] Key type lgssc unregistered [ 3437.673329] LNet: 168830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3437.675532] LNetError: 168830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3437.683275] LNet: Removed LNI 192.168.201.119@tcp [ 3437.990141] Key type .llcrypt unregistered [ 3437.991118] Key type ._llcrypt unregistered [ 3446.080959] Key type ._llcrypt registered [ 3446.081728] Key type .llcrypt registered [ 3446.113597] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3446.369985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3446.404029] alg: No test for adler32 (adler32-zlib) [ 3447.238360] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 3447.323047] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3448.895123] Key type lgssc registered [ 3449.209755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3451.781833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3451.786672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.911798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3454.245900] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3457.081730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3457.197210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3458.566724] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3461.002550] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3462.863343] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3465.136311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3465.253041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3467.252741] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3469.770120] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3478.566842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3482.591762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3482.593829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3482.597350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3483.618023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3483.618058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3483.618095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3483.619824] Lustre: Skipped 1 previous similar message [ 3487.819808] Lustre: server umount lustre-OST0000 complete [ 3493.344027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3493.348037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3493.355378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3495.202110] Lustre: server umount lustre-MDT0000 complete [ 3496.499343] LustreError: 172708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394457 with bad export cookie 10241888711313429359 [ 3496.503991] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3496.505870] LustreError: 172708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3496.623737] Lustre: server umount lustre-MDT0001 complete [ 3500.136330] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3501.217188] Key type lgssc unregistered [ 3501.341379] LNet: 173761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3501.344510] LNetError: 173761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3501.354444] LNet: Removed LNI 192.168.201.119@tcp [ 3501.739115] Key type .llcrypt unregistered [ 3501.740453] Key type ._llcrypt unregistered [ 3513.871816] Key type ._llcrypt registered [ 3513.872935] Key type .llcrypt registered [ 3513.910991] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3514.286259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3514.343154] alg: No test for adler32 (adler32-zlib) [ 3515.204476] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 3515.298935] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3516.879135] Key type lgssc registered [ 3517.254975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3519.740603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3519.746347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.890161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3522.191200] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3524.917797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.050545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3526.391265] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3528.882474] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3530.734410] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3532.941236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3533.059586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.148764] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3537.548241] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3546.406602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3550.688253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3550.691458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.695656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.712788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3551.715085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.715237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.720829] Lustre: Skipped 1 previous similar message [ 3555.536092] Lustre: server umount lustre-OST0000 complete [ 3556.832449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3556.833280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3556.838367] Lustre: Skipped 1 previous similar message [ 3561.952097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3561.953819] Lustre: Skipped 2 previous similar messages [ 3567.072180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3567.074168] Lustre: Skipped 1 previous similar message [ 3571.167129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3571.230443] Lustre: server umount lustre-MDT0000 complete [ 3572.191950] LustreError: 175433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3572.196523] LustreError: 175433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3572.439799] LustreError: 176169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394533 with bad export cookie 798699325116572805 [ 3572.442843] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3572.444143] LustreError: 176169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3572.563488] Lustre: server umount lustre-MDT0001 complete [ 3576.094095] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3577.040266] Key type lgssc unregistered [ 3577.176360] LNet: 178986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3577.179176] LNetError: 178986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3577.185318] LNet: Removed LNI 192.168.201.119@tcp [ 3577.481415] Key type .llcrypt unregistered [ 3577.482583] Key type ._llcrypt unregistered [ 3585.808522] Key type ._llcrypt registered [ 3585.809429] Key type .llcrypt registered [ 3585.852393] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3586.276240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3586.282635] alg: No test for adler32 (adler32-zlib) [ 3587.127505] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 3587.212945] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3588.791195] Key type lgssc registered [ 3589.154143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3592.565687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3592.571602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.693471] Lustre: *** cfs_fail_loc=135, val=0*** [ 3593.730512] LustreError: 180482:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3593.732316] LustreError: 180482:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.119@tcp: cfg command failed: rc = -2 [ 3593.734672] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3593.734672] [ 3593.737057] LustreError: MGC192.168.201.119@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3593.744303] LustreError: 180444:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3593.750121] LustreError: 180444:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3593.752467] LustreError: 180444:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3593.754080] LustreError: 180444:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3593.785546] Lustre: server umount lustre-MDT0000 complete [ 3593.786978] LustreError: 180444:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3595.906343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.043098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3597.322705] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3600.097700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.232730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3601.631932] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3604.120894] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3606.024534] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3609.761191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3609.762218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3609.766597] Lustre: Skipped 1 previous similar message [ 3609.771055] Lustre: Skipped 1 previous similar message [ 3612.813520] Lustre: server umount lustre-MDT0000 complete [ 3613.896104] LustreError: 180836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394575 with bad export cookie 13382526447693165063 [ 3613.899217] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3613.901033] LustreError: 180836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3614.012980] Lustre: server umount lustre-MDT0001 complete [ 3617.453090] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3618.415789] Key type lgssc unregistered [ 3618.535369] LNet: 183170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3618.537760] LNetError: 183170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3618.543302] LNet: Removed LNI 192.168.201.119@tcp [ 3618.834713] Key type .llcrypt unregistered [ 3618.835827] Key type ._llcrypt unregistered [ 3625.664540] Key type ._llcrypt registered [ 3625.665460] Key type .llcrypt registered [ 3625.701084] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3625.795168] Key type .llcrypt unregistered [ 3625.796058] Key type ._llcrypt unregistered [ 3632.578231] Key type ._llcrypt registered [ 3632.579252] Key type .llcrypt registered [ 3632.617761] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3633.118173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3633.138937] alg: No test for adler32 (adler32-zlib) [ 3633.996515] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 3634.094958] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3635.679160] Key type lgssc registered [ 3636.045105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3638.566235] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3638.571881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.983367] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3659.103240] LustreError: 185270:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 3685.727174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3687.748291] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3690.190634] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3692.455301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.609818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.631729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3693.860331] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3696.510627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3696.630543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3697.850973] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3699.963556] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3701.734782] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3702.654593] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3707.872121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3707.874406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3707.878824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3710.948314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3710.948882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.492230] Lustre: server umount lustre-OST0000 complete [ 3716.064556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3716.065141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3716.069276] Lustre: Skipped 1 previous similar message [ 3716.071076] Lustre: Skipped 1 previous similar message [ 3720.786431] Lustre: server umount lustre-MDT0000 complete [ 3721.183857] LustreError: 186310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.187908] LustreError: 186310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3721.914116] LustreError: 187939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394683 with bad export cookie 13738101175071022173 [ 3721.917123] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3721.917236] LustreError: 187939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3722.014844] Lustre: server umount lustre-MDT0001 complete [ 3725.217316] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3726.130387] Key type lgssc unregistered [ 3726.262104] LNet: 189191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3726.263995] LNetError: 189191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3726.269210] LNet: Removed LNI 192.168.201.119@tcp [ 3726.543102] Key type .llcrypt unregistered [ 3726.543948] Key type ._llcrypt unregistered [ 3735.015906] Key type ._llcrypt registered [ 3735.016832] Key type .llcrypt registered [ 3735.053968] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3735.417699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3735.497772] alg: No test for adler32 (adler32-zlib) [ 3736.355507] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 3736.448683] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3738.039201] Key type lgssc registered [ 3738.405942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3740.930797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3740.939165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.075861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3743.381855] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3746.114707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3746.234576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3747.584440] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3749.959395] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3751.719345] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3753.918158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3754.042082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.230677] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3758.749465] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3767.398719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3770.481033] Lustre: Failing over lustre-MDT0000 [ 3770.572239] Lustre: server umount lustre-MDT0000 complete [ 3771.872403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3771.875152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.880572] LustreError: 192792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3771.885739] LustreError: 192792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3772.896529] LustreError: 190575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.897816] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.902623] LustreError: 190575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3772.906606] Lustre: Skipped 1 previous similar message [ 3774.441540] LustreError: 193075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3778.016363] LustreError: 192792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3778.021569] LustreError: 192792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3783.136266] LustreError: 190575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3783.142459] LustreError: 190575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3783.535174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.574994] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3783.664578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.700742] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3784.675369] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3785.063513] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3786.599963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3788.774041] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3788.782323] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3788.800741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3794.965138] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3802.591829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.594978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.599607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3804.130276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3804.130693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3804.135677] Lustre: Skipped 1 previous similar message [ 3807.175738] Lustre: server umount lustre-OST0000 complete [ 3809.248509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3809.248732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3809.249340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3809.255039] Lustre: Skipped 2 previous similar messages [ 3814.368148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3814.370148] Lustre: Skipped 2 previous similar messages [ 3814.428692] Lustre: server umount lustre-MDT0000 complete [ 3815.642225] LustreError: 190555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394776 with bad export cookie 8596416483973873021 [ 3815.645573] LustreError: 190555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3815.646325] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3815.757858] Lustre: server umount lustre-MDT0001 complete [ 3819.280900] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3820.406652] Key type lgssc unregistered [ 3820.542324] LNet: 195617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3820.544533] LNetError: 195617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3820.554275] LNet: Removed LNI 192.168.201.119@tcp [ 3820.898539] Key type .llcrypt unregistered [ 3820.899585] Key type ._llcrypt unregistered [ 3829.279268] Key type ._llcrypt registered [ 3829.280333] Key type .llcrypt registered [ 3829.319495] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3829.743874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3829.755616] alg: No test for adler32 (adler32-zlib) [ 3830.600303] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 3830.688980] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3832.263115] Key type lgssc registered [ 3832.614109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3835.124667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3835.131538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3836.294563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3837.588740] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3840.464078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.586103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3841.938222] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3844.350727] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3846.113847] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3848.275895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3848.388828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3850.252747] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3852.593719] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3861.735065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3863.620390] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3867.078902] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3878.778301] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3887.586121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3887.586175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3887.593287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3887.597487] Lustre: Skipped 1 previous similar message [ 3892.705509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3892.707268] Lustre: Skipped 1 previous similar message [ 3893.127416] Lustre: server umount lustre-OST0000 complete [ 3896.799859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3896.801929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3896.805012] Lustre: Skipped 1 previous similar message [ 3896.806700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3897.825255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.379306] Lustre: server umount lustre-MDT0000 complete [ 3901.667544] LustreError: 199552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394862 with bad export cookie 5031671386914275436 [ 3901.672917] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3901.798364] Lustre: server umount lustre-MDT0001 complete [ 3905.147574] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3906.126273] Key type lgssc unregistered [ 3906.252301] LNet: 200862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3906.254213] LNetError: 200862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3906.264247] LNet: Removed LNI 192.168.201.119@tcp [ 3906.547249] Key type .llcrypt unregistered [ 3906.548466] Key type ._llcrypt unregistered [ 3913.305732] Key type ._llcrypt registered [ 3913.306549] Key type .llcrypt registered [ 3913.339737] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3913.635039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3913.657116] alg: No test for adler32 (adler32-zlib) [ 3914.501393] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 3914.577645] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3916.151140] Key type lgssc registered [ 3916.557566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3919.295156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3919.301098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3920.449163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3921.725031] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3924.457374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3924.574114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3925.793368] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3927.900690] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3929.520643] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3931.664407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3931.772615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3933.500981] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3935.625937] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3946.150073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3952.508378] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3962.816256] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3973.413929] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3982.409096] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3990.495837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3990.498031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3990.501754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3992.546168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3992.546864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3992.546887] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3997.668251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3997.670635] Lustre: Skipped 2 previous similar messages [ 4002.784237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4002.786555] Lustre: Skipped 1 previous similar message [ 4003.295158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4003.345959] Lustre: server umount lustre-OST0000 complete [ 4006.367783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4006.370374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.904539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4007.905907] Lustre: Skipped 1 previous similar message [ 4010.578485] Lustre: server umount lustre-MDT0000 complete [ 4011.715205] LustreError: 202127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394972 with bad export cookie 5674628448352903188 [ 4011.717787] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4011.719023] LustreError: 202127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4011.828750] Lustre: server umount lustre-MDT0001 complete [ 4014.970446] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 4015.896303] Key type lgssc unregistered [ 4016.010311] LNet: 207719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4016.012958] LNetError: 207719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4016.020259] LNet: Removed LNI 192.168.201.119@tcp [ 4016.300650] Key type .llcrypt unregistered [ 4016.301521] Key type ._llcrypt unregistered [ 4024.399401] Key type ._llcrypt registered [ 4024.400547] Key type .llcrypt registered [ 4024.431765] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4024.725651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4024.753057] alg: No test for adler32 (adler32-zlib) [ 4025.588512] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4025.669326] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 4027.239121] Key type lgssc registered [ 4027.551167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4029.812679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4029.817168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.942808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4032.195847] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4034.737852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4034.848265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4036.041813] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4038.141226] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4039.724892] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4041.796972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4041.914648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.620489] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4045.766864] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4056.236694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4065.760266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4065.763031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.768713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4066.784867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4066.785719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4066.787631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4070.474452] Lustre: server umount lustre-OST0000 complete [ 4071.904238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.904832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4071.908539] Lustre: Skipped 1 previous similar message [ 4071.911752] Lustre: Skipped 2 previous similar messages [ 4077.024276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.027027] Lustre: Skipped 1 previous similar message [ 4077.783048] Lustre: server umount lustre-MDT0000 complete [ 4079.054434] LustreError: 210947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395040 with bad export cookie 6191006660278757037 [ 4079.058234] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4079.059641] LustreError: 210947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4079.187794] Lustre: server umount lustre-MDT0001 complete [ 4082.961433] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 4083.912558] Key type lgssc unregistered [ 4084.045508] LNet: 212800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4084.048247] LNetError: 212800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4084.067307] LNet: Removed LNI 192.168.201.119@tcp [ 4084.376738] Key type .llcrypt unregistered [ 4084.378314] Key type ._llcrypt unregistered [ 4091.466234] Key type ._llcrypt registered [ 4091.467841] Key type .llcrypt registered [ 4091.503352] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4091.798491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4091.816156] alg: No test for adler32 (adler32-zlib) [ 4092.657446] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4092.733765] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 4094.311168] Key type lgssc registered [ 4094.643941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4097.088356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4097.093296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4098.221950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4099.513542] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4102.128895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4102.238441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4103.503124] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4105.774402] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4107.442196] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4109.450608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4109.552881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4111.317256] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4113.714454] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4123.874496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4128.224193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4128.226638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4128.230665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4129.248402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4129.249702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4129.250383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4129.256597] Lustre: Skipped 1 previous similar message [ 4133.453193] Lustre: server umount lustre-OST0000 complete [ 4137.952055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4137.954424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.958500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.608168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.610730] Lustre: Skipped 3 previous similar messages [ 4149.215084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4149.277370] Lustre: server umount lustre-MDT0000 complete [ 4149.727920] LustreError: 215475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4149.733851] LustreError: 215475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4150.466087] LustreError: 214017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395111 with bad export cookie 10122982316979614903 [ 4150.469115] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4150.470281] LustreError: 214017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4150.588766] Lustre: server umount lustre-MDT0001 complete [ 4154.264865] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 4155.323858] Key type lgssc unregistered [ 4155.455347] LNet: 217740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4155.458482] LNetError: 217740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4155.470309] LNet: Removed LNI 192.168.201.119@tcp [ 4155.819718] Key type .llcrypt unregistered [ 4155.820915] Key type ._llcrypt unregistered [ 4163.315622] Key type ._llcrypt registered [ 4163.316640] Key type .llcrypt registered [ 4163.354170] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4163.796538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4163.804582] alg: No test for adler32 (adler32-zlib) [ 4164.660768] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4164.756551] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 4166.335166] Key type lgssc registered [ 4166.711386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4169.407721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4169.414053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4170.560259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.927976] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4174.737869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.880680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4176.320391] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4178.780212] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4180.627410] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4182.961167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4183.094995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.075249] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4187.588696] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4196.390917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4200.976684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4201.006778] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4201.096783] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4203.129125] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4205.515527] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4206.014511] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4216.569569] Lustre: Permanently deactivating lustre-OST0001 [ 4216.576757] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4218.036776] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4227.259724] Lustre: Permanently reactivating lustre-OST0001 [ 4227.265497] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4227.267469] Lustre: Skipped 2 previous similar messages [ 4235.260127] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.264112] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4235.266408] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4235.269687] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4235.281352] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4240.352157] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4240.352371] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4240.354589] LustreError: Skipped 1 previous similar message [ 4240.359811] Lustre: Skipped 1 previous similar message [ 4240.360782] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4240.366382] Lustre: Skipped 1 previous similar message [ 4242.493760] Lustre: server umount lustre-OST0001 complete [ 4245.987618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4245.988401] LustreError: 221171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4245.989123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4245.990940] Lustre: Skipped 2 previous similar messages [ 4245.995228] LustreError: 221171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4249.801475] Lustre: server umount lustre-OST0000 complete [ 4251.104429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.105144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4251.108641] Lustre: Skipped 1 previous similar message [ 4251.110549] Lustre: Skipped 1 previous similar message [ 4256.224043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4256.225955] Lustre: Skipped 2 previous similar messages [ 4257.173200] Lustre: server umount lustre-MDT0000 complete [ 4258.358304] LustreError: 218958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395219 with bad export cookie 4350234649496329522 [ 4258.361176] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4258.363586] LustreError: 218958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4258.468409] Lustre: server umount lustre-MDT0001 complete [ 4261.995348] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 4262.973702] Key type lgssc unregistered [ 4263.096362] LNet: 224034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4263.098589] LNetError: 224034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4263.105258] LNet: Removed LNI 192.168.201.119@tcp [ 4263.381789] Key type .llcrypt unregistered [ 4263.382721] Key type ._llcrypt unregistered [ 4277.117840] Key type ._llcrypt registered [ 4277.118913] Key type .llcrypt registered [ 4277.166777] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4277.611160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4277.626803] alg: No test for adler32 (adler32-zlib) [ 4278.480632] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4278.565740] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 4280.143097] Key type lgssc registered [ 4280.616663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4283.048417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4283.055457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4284.140914] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4284.148066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4284.211653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4285.532222] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4288.148069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4288.184705] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4288.195335] Lustre: 226211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4288.290768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4289.608774] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4291.882708] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4293.575917] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4295.750966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4295.821063] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4295.889317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4297.762128] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4300.083326] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4300.305707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 4306.796851] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4309.243203] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4318.815448] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4327.379884] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4335.918413] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4353.084854] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4353.089467] Lustre: Skipped 1 previous similar message [ 4385.341776] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4385.344739] Lustre: Skipped 3 previous similar messages [ 4455.678250] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4455.680789] Lustre: Skipped 9 previous similar messages [ 4469.220792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4469.221191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.223956] LustreError: Skipped 1 previous similar message [ 4469.224659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4469.229484] Lustre: Skipped 1 previous similar message [ 4469.235083] Lustre: Skipped 1 previous similar message [ 4471.246584] Lustre: server umount lustre-OST0000 complete [ 4472.800019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4472.802967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.808798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4474.336171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4474.337178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.611811] Lustre: server umount lustre-MDT0000 complete [ 4479.853204] LustreError: 226202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395441 with bad export cookie 4675803923182834495 [ 4479.856295] LustreError: 226202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4479.856540] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4479.978072] Lustre: server umount lustre-MDT0001 complete [ 4483.591477] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 4484.601552] Key type lgssc unregistered [ 4484.726311] LNet: 231015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4484.729168] LNetError: 231015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4484.738073] LNet: Removed LNI 192.168.201.119@tcp [ 4485.056872] Key type .llcrypt unregistered [ 4485.058828] Key type ._llcrypt unregistered [ 4494.023754] Key type ._llcrypt registered [ 4494.024773] Key type .llcrypt registered [ 4494.067583] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4494.415275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4494.429091] alg: No test for adler32 (adler32-zlib) [ 4495.310209] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4495.416996] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 4496.999154] Key type lgssc registered [ 4497.377274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4500.012630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4500.019067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4501.160536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4502.509968] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4505.340067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.470310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4506.797163] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4509.232786] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4511.064546] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4513.289288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4513.421050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4515.388257] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4517.774110] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4526.755474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 4536.288389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4536.290899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4536.296224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4537.313113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4537.313181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4537.314977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4539.979859] Lustre: server umount lustre-OST0000 complete [ 4541.407790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4541.409614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4541.412851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4541.414610] Lustre: Skipped 1 previous similar message [ 4547.342782] Lustre: server umount lustre-MDT0000 complete [ 4547.551952] LustreError: 235155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4547.556019] LustreError: 235155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4548.599385] LustreError: 232382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395509 with bad export cookie 8507325784600296112 [ 4548.604606] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4548.742835] Lustre: server umount lustre-MDT0001 complete [ 4552.520806] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 4553.637090] Key type lgssc unregistered [ 4553.771561] LNet: 236108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4553.774645] LNetError: 236108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4553.781455] LNet: Removed LNI 192.168.201.119@tcp [ 4554.117827] Key type .llcrypt unregistered [ 4554.119087] Key type ._llcrypt unregistered [ 4562.972659] Key type ._llcrypt registered [ 4562.973618] Key type .llcrypt registered [ 4563.017199] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4563.549962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4563.563955] alg: No test for adler32 (adler32-zlib) [ 4564.426568] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4564.531515] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 4566.119140] Key type lgssc registered [ 4566.522808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4833.587845] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 4834.562582] Key type lgssc unregistered [ 4834.686282] LNet: 238101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4834.688475] LNetError: 238101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4834.699293] LNet: Removed LNI 192.168.201.119@tcp [ 4834.988418] Key type .llcrypt unregistered [ 4834.989516] Key type ._llcrypt unregistered [ 4842.946824] Key type ._llcrypt registered [ 4842.947850] Key type .llcrypt registered [ 4842.983113] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 4849.274274] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4849.597753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4849.615214] alg: No test for adler32 (adler32-zlib) [ 4850.475321] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4850.565198] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 4852.143163] Key type lgssc registered [ 4852.497340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4862.193657] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4552 sec ========== 00:10:23 (1790395823) [ 4862.717527] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:10:24 (1790395824) === [ 4863.798200] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:10:25 (1790395825) === [ 4876.896100] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 4877.934704] Key type lgssc unregistered [ 4878.058318] LNet: 243300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4878.060551] LNetError: 243300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4878.067360] LNet: Removed LNI 192.168.201.119@tcp [ 4878.416355] Key type .llcrypt unregistered [ 4878.417520] Key type ._llcrypt unregistered