[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478008625 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003241] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008396] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011175] LSM: Security Framework initializing [ 0.012064] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014072] *** VALIDATE selinux *** [ 0.022258] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026229] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] *** VALIDATE tmpfs *** [ 0.031488] *** VALIDATE proc *** [ 0.033182] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036031] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043554] debug: unmapping init [mem 0xffffffffa0459000-0xffffffffa0460fff] [ 0.045183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046753] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047028] ... version: 2 [ 0.048010] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054303] rcu: Hierarchical SRCU implementation. [ 0.056482] smp: Bringing up secondary CPUs ... [ 0.057603] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.061135] smp: Brought up 1 node, 4 CPUs [ 0.063011] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130264] node 0 deferred pages initialised in 62ms [ 0.133490] devtmpfs: initialized [ 0.135271] x86/mm: Memory block size: 128MB [ 0.137755] gcov: version magic: 0x41383552 [ 0.140203] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145293] pinctrl core: initialized pinctrl subsystem [ 0.146134] [ 0.146487] ************************************************************* [ 0.148009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149007] ** ** [ 0.151012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153009] ** ** [ 0.154007] ** This means that this kernel is built to expose internal ** [ 0.156011] ** IOMMU data structures, which may compromise security on ** [ 0.157008] ** your system. ** [ 0.158007] ** ** [ 0.159009] ** If you see this message and you are not debugging the ** [ 0.161008] ** kernel, report this immediately to your vendor! ** [ 0.162007] ** ** [ 0.164015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165013] ************************************************************* [ 0.167717] NET: Registered protocol family 16 [ 0.169328] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175116] cpuidle: using governor menu [ 0.176914] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178341] PCI: Using configuration type 1 for base access [ 0.180119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.188151] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.191020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.194157] cryptd: max_cpu_qlen set to 1000 [ 0.197820] ACPI: Added _OSI(Module Device) [ 0.199012] ACPI: Added _OSI(Processor Device) [ 0.201011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206985] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213031] ACPI: Interpreter enabled [ 0.214056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215018] ACPI: Using IOAPIC for interrupt routing [ 0.217142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.221445] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.229158] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.234018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237065] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.241454] acpiphp: Slot [2] registered [ 0.243147] acpiphp: Slot [5] registered [ 0.245127] acpiphp: Slot [6] registered [ 0.247111] acpiphp: Slot [7] registered [ 0.248106] acpiphp: Slot [8] registered [ 0.250165] acpiphp: Slot [9] registered [ 0.251118] acpiphp: Slot [10] registered [ 0.253131] acpiphp: Slot [3] registered [ 0.254158] acpiphp: Slot [4] registered [ 0.256116] acpiphp: Slot [11] registered [ 0.257079] acpiphp: Slot [12] registered [ 0.258062] acpiphp: Slot [13] registered [ 0.259019] acpiphp: Slot [14] registered [ 0.259953] acpiphp: Slot [15] registered [ 0.262132] acpiphp: Slot [16] registered [ 0.263098] acpiphp: Slot [17] registered [ 0.265078] acpiphp: Slot [18] registered [ 0.266082] acpiphp: Slot [19] registered [ 0.266984] acpiphp: Slot [20] registered [ 0.267064] acpiphp: Slot [21] registered [ 0.268065] acpiphp: Slot [22] registered [ 0.270110] acpiphp: Slot [23] registered [ 0.271078] acpiphp: Slot [24] registered [ 0.273094] acpiphp: Slot [25] registered [ 0.274088] acpiphp: Slot [26] registered [ 0.275118] acpiphp: Slot [27] registered [ 0.277097] acpiphp: Slot [28] registered [ 0.278148] acpiphp: Slot [29] registered [ 0.280094] acpiphp: Slot [30] registered [ 0.281096] acpiphp: Slot [31] registered [ 0.283080] PCI host bridge to bus 0000:00 [ 0.284019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.286024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.288027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.293021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.296026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.297214] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.300096] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.303393] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.313862] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.319530] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.321016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.322020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.325028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.327575] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.329628] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.332076] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.335042] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.340017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.352015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.356984] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.363899] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.393018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.429025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.441042] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.449015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.456018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.480015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.491341] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.503017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.519027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.534027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.542000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.551015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.555017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.576014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.588799] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.596019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.609017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.630018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.642812] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.651017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.659014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.674015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.682187] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.683388] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.686311] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.687312] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.689179] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.693188] iommu: Default domain type: Passthrough [ 0.695494] SCSI subsystem initialized [ 0.696118] ACPI: bus type USB registered [ 0.698075] usbcore: registered new interface driver usbfs [ 0.699056] usbcore: registered new interface driver hub [ 0.701056] usbcore: registered new device driver usb [ 0.702171] pps_core: LinuxPPS API ver. 1 registered [ 0.704010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.706046] PTP clock support registered [ 0.707106] EDAC MC: Ver: 3.0.0 [ 0.709117] PCI: Using ACPI for IRQ routing [ 0.710848] NetLabel: Initializing [ 0.712011] NetLabel: domain hash size = 128 [ 0.713007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.715100] NetLabel: unlabeled traffic allowed by default [ 0.716373] vgaarb: loaded [ 0.718326] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.719012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.726660] clocksource: Switched to clocksource kvm-clock [ 0.826506] VFS: Disk quotas dquot_6.6.0 [ 0.828070] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.830548] *** VALIDATE ramfs *** [ 0.831914] *** VALIDATE hugetlbfs *** [ 0.833333] pnp: PnP ACPI init [ 0.835505] pnp: PnP ACPI: found 6 devices [ 0.851810] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.854494] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.855996] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.858295] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.860675] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.862834] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.865716] NET: Registered protocol family 2 [ 0.867762] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.871888] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.874791] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.880725] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.883888] TCP: Hash tables configured (established 65536 bind 65536) [ 0.886800] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.889598] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.891924] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.894540] NET: Registered protocol family 1 [ 0.897300] RPC: Registered named UNIX socket transport module. [ 0.900227] RPC: Registered udp transport module. [ 0.901680] RPC: Registered tcp transport module. [ 0.902988] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.905300] NET: Registered protocol family 44 [ 0.906782] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.909332] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.911466] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.914168] PCI: CLS 0 bytes, default 64 [ 0.916320] Unpacking initramfs... [ 2.232434] debug: unmapping init [mem 0xffff8acbbcc54000-0xffff8acbbffbffff] [ 2.234805] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.236100] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.238371] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.729064] Initialise system trusted keyrings [ 2.730748] Key type blacklist registered [ 2.732940] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.742238] zbud: loaded [ 2.746248] *** VALIDATE nfs *** [ 2.747113] *** VALIDATE nfs4 *** [ 2.748942] pstore: using deflate compression [ 2.755189] Platform Keyring initialized [ 2.856354] NET: Registered protocol family 38 [ 2.858673] Key type asymmetric registered [ 2.860316] Asymmetric key parser 'x509' registered [ 2.862121] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.864839] io scheduler mq-deadline registered [ 2.866551] io scheduler kyber registered [ 2.868355] io scheduler bfq registered [ 2.870180] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.873107] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.876029] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.878244] ACPI: Power Button [PWRF] [ 2.882228] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.887521] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.897250] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.902553] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.914372] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.940305] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.965442] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.969298] Non-volatile memory driver v1.3 [ 2.970406] Linux agpgart interface v0.103 [ 2.994371] virtio_blk virtio1: [vda] 146864 512-byte logical blocks (75.2 MB/71.7 MiB) [ 2.996244] vda: detected capacity change from 0 to 75194368 [ 3.008467] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.010482] vdb: detected capacity change from 0 to 1073741824 [ 3.021390] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.023293] vdc: detected capacity change from 0 to 2621440000 [ 3.036518] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.039268] vdd: detected capacity change from 0 to 2621440000 [ 3.050582] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.052595] vde: detected capacity change from 0 to 4294967296 [ 3.067175] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.068763] vdf: detected capacity change from 0 to 4294967296 [ 3.076150] libphy: Fixed MDIO Bus: probed [ 3.081781] usbcore: registered new interface driver usbserial_generic [ 3.083750] usbserial: USB Serial support registered for generic [ 3.086122] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.090545] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.092460] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.094658] mousedev: PS/2 mouse device common for all mice [ 3.096895] rtc_cmos 00:05: RTC can wake from S4 [ 3.099222] rtc_cmos 00:05: registered as rtc0 [ 3.101373] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.102181] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.103565] intel_pstate: CPU model not supported [ 3.108765] hid: raw HID events driver (C) Jiri Kosina [ 3.114408] usbcore: registered new interface driver usbhid [ 3.115862] usbhid: USB HID core driver [ 3.117370] drop_monitor: Initializing network drop monitor service [ 3.117739] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.119210] Initializing XFRM netlink socket [ 3.122995] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.123080] NET: Registered protocol family 10 [ 3.127490] Segment Routing with IPv6 [ 3.128368] NET: Registered protocol family 17 [ 3.129661] mpls_gso: MPLS GSO support [ 3.134416] RAS: Correctable Errors collector initialized. [ 3.136469] AVX version of gcm_enc/dec engaged. [ 3.138221] AES CTR mode by8 optimization enabled [ 3.212140] sched_clock: Marking stable (3212076176, 0)->(4043192827, -831116651) [ 3.215529] registered taskstats version 1 [ 3.217748] Loading compiled-in X.509 certificates [ 3.220314] zswap: loaded using pool lzo/zbud [ 3.240167] Key type big_key registered [ 3.249627] Key type encrypted registered [ 3.251650] ima: No TPM chip found, activating TPM-bypass! [ 3.252956] ima: Allocated hash algorithm: sha1 [ 3.253889] ima: No architecture policies found [ 3.255263] evm: Initialising EVM extended attributes: [ 3.256402] evm: security.selinux [ 3.257190] evm: security.ima [ 3.257905] evm: security.capability [ 3.258734] evm: HMAC attrs: 0x1 [ 3.260751] rtc_cmos 00:05: setting system clock to 2026-08-27 17:48:07 UTC (1787852887) [ 3.266590] debug: unmapping init [mem 0xffffffffa1403000-0xffffffffa15fffff] [ 3.268653] debug: unmapping init [mem 0xffffffffa0182000-0xffffffffa0458fff] [ 3.276106] Write protecting the kernel read-only data: 28672k [ 3.278633] debug: unmapping init [mem 0xffffffff9e803000-0xffffffff9e9fffff] [ 3.280467] debug: unmapping init [mem 0xffffffff9f114000-0xffffffff9f1fffff] [ 3.308640] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.314613] systemd[1]: Detected virtualization kvm. [ 3.316057] systemd[1]: Detected architecture x86-64. [ 3.317337] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.341089] systemd[1]: No hostname configured. [ 3.342796] systemd[1]: Set hostname to . [ 3.344904] random: systemd: uninitialized urandom read (16 bytes read) [ 3.347361] systemd[1]: Initializing machine ID from random generator. [ 3.460770] random: systemd: uninitialized urandom read (16 bytes read) [ 3.463788] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.469785] random: systemd: uninitialized urandom read (16 bytes read) [ 3.471454] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.476590] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.037746] device-mapper: uevent: version 1.0.3 [ 4.039237] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK [ 4.644655] random: fast init done ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.666739] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 4.714766] scsi host0: ata_piix [ 4.733446] scsi host1: ata_piix [ 4.735991] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.739182] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.329706] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.770658] random: crng init done [ 9.772097] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.987323] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.054876] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.252248] SELinux: Disabled at runtime. [ 11.306241] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.313644] systemd[1]: Detected virtualization kvm. [ 11.315289] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.691167] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.694709] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.698270] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.701344] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.703670] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.713233] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.716379] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 11.851217] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.032604] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.307408] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.392194] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.429196] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.442713] EDAC sbridge: Ver: 1.1.2 [ 13.679050] Key type dns_resolver registered [ 14.002534] NFS: Registering the id_resolver key type [ 14.004388] Key type id_resolver registered [ 14.006115] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg117-server login: [ 38.945381] libcfs: loading out-of-tree module taints kernel. [ 38.966229] Key type ._llcrypt registered [ 38.967662] Key type .llcrypt registered [ 39.012240] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 54.396469] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 56.600401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 56.613793] alg: No test for adler32 (adler32-zlib) [ 58.144716] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 59.072778] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 61.039594] Key type lgssc registered [ 63.075740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.002774] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 88.419011] hrtimer: interrupt took 4018488 ns [ 121.090856] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 134.723689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 134.800257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 136.085386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 136.134360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 136.238394] Lustre: lustre-MDT0000: new disk, initializing [ 136.337812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 136.359369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 140.594912] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.040564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.180573] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 155.210568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 155.215640] Lustre: Skipped 1 previous similar message [ 155.337493] Lustre: lustre-MDT0001: new disk, initializing [ 155.406154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 155.440406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 155.451997] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 159.751643] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.810838] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 175.287841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 175.539669] Lustre: lustre-OST0000: new disk, initializing [ 175.543707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 175.547957] Lustre: 8442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 175.614370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.829440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 181.853956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 181.977703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 182.722582] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.103332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.335263] Lustre: lustre-OST0001: new disk, initializing [ 199.342503] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 199.351734] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.426434] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.821321] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.417587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 207.422601] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 207.509550] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 219.216518] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.514657] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 232.630737] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.717492] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing yml_node [ 242.774554] Lustre: DEBUG MARKER: Client: 2.17.57.84 [ 245.474448] Lustre: DEBUG MARKER: MDS: 2.17.57.84 [ 247.795839] Lustre: DEBUG MARKER: OSS: 2.17.57.84 [ 249.491948] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 13:52:12 EDT 2026 [ 266.895405] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 268.751724] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 288.736339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 288.739505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.776365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.248307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.265127] Lustre: Skipped 1 previous similar message [ 292.223801] Lustre: server umount lustre-MDT0000 complete [ 296.224254] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853180 with bad export cookie 15836925606885409161 [ 296.228869] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 296.241865] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 299.488382] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.489170] LustreError: 9752:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.495653] Lustre: Skipped 1 previous similar message [ 299.496874] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.496881] Lustre: Skipped 4 previous similar messages [ 299.550551] LustreError: 9752:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 301.535971] LustreError: 6513:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.543528] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 301.554765] LustreError: 6513:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 301.571509] Lustre: Skipped 2 previous similar messages [ 302.878533] Lustre: server umount lustre-MDT0001 complete [ 307.531642] Lustre: server umount lustre-OST0000 complete [ 311.852356] Lustre: server umount lustre-OST0001 complete [ 318.228765] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 329.430735] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 366.801332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.071674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 367.101780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 367.186879] Lustre: lustre-MDT0000: new disk, initializing [ 367.275641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 367.292589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 372.812022] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 384.938744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.129119] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 385.216425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 385.219886] Lustre: Skipped 1 previous similar message [ 385.332271] Lustre: lustre-MDT0001: new disk, initializing [ 385.440485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 385.483107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 385.503492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 390.587386] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 401.126562] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 407.282304] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 414.524528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.856895] Lustre: lustre-OST0000: new disk, initializing [ 414.860511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 414.868301] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 414.973754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.970477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 416.998507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 417.187533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 422.227242] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 432.529223] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 438.396668] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 438.744356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 443.633727] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 443.846576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 447.979182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.981420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 447.984059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 448.007187] Lustre: Skipped 2 previous similar messages [ 453.091737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 453.095604] Lustre: Skipped 2 previous similar messages [ 460.256913] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 460.580941] Lustre: server umount lustre-OST0000 complete [ 467.425697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 467.445704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.487922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.503751] Lustre: Skipped 2 previous similar messages [ 471.415778] Lustre: server umount lustre-MDT0000 complete [ 473.570604] LustreError: 15081:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.570604] LustreError: 15959:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.140018] LustreError: 15871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853360 with bad export cookie 15836925606885410694 [ 476.154768] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 476.173249] LustreError: 15871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 476.677657] Lustre: server umount lustre-MDT0001 complete [ 480.740236] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 13:56:03 (1787853363) [ 488.992465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.800080] LustreError: 19311:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.960533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.561358] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 500.194970] LustreError: 19312:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.217519] LustreError: 19312:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 504.501035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.458856] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 518.722309] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 524.508582] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 531.794053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 539.188663] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 547.472764] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 559.019490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.022306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 559.034238] Lustre: Skipped 2 previous similar messages [ 559.063777] Lustre: Skipped 3 previous similar messages [ 572.383141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 572.692433] Lustre: server umount lustre-OST0000 complete [ 579.552718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.564258] Lustre: Skipped 1 previous similar message [ 591.327161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 591.642438] Lustre: server umount lustre-MDT0000 complete [ 594.912603] LustreError: 19311:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.942044] LustreError: 19311:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 595.687038] LustreError: 19292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853479 with bad export cookie 15836925606885411576 [ 595.706355] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.241788] Lustre: server umount lustre-MDT0001 complete [ 608.744860] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 612.326451] Key type lgssc unregistered [ 612.815065] LNet: 22849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.818773] LNetError: 22849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.841725] LNet: Removed LNI 192.168.201.117@tcp [ 613.923787] Key type .llcrypt unregistered [ 613.925536] Key type ._llcrypt unregistered [ 639.286897] Key type ._llcrypt registered [ 639.288924] Key type .llcrypt registered [ 639.445059] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 640.605944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 640.727698] alg: No test for adler32 (adler32-zlib) [ 641.879185] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 642.179353] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 643.919192] Key type lgssc registered [ 645.479976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 653.287723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 653.309312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.797081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 659.115938] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 664.053718] LustreError: 24268:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.088074] LustreError: 24268:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 666.875925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.132622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 672.101977] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 682.225813] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 690.767766] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 696.901491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 697.409363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.458228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 703.579624] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 710.191525] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 728.053657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 728.068753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 728.087052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 728.093755] Lustre: Skipped 1 previous similar message [ 732.546205] Lustre: server umount lustre-OST0000 complete [ 738.272797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 738.279818] Lustre: Skipped 1 previous similar message [ 738.282646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.561475] Lustre: server umount lustre-MDT0000 complete [ 743.401838] LustreError: 24268:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 743.421564] LustreError: 24268:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 746.152517] LustreError: 24247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853630 with bad export cookie 9987354883728787239 [ 746.153715] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 746.166073] LustreError: 24247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 746.582930] Lustre: server umount lustre-MDT0001 complete [ 758.767121] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 762.396918] Key type lgssc unregistered [ 762.846141] LNet: 28247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.853742] LNetError: 28247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.883302] LNet: Removed LNI 192.168.201.117@tcp [ 764.983670] Key type .llcrypt unregistered [ 764.985558] Key type ._llcrypt unregistered [ 790.013780] Key type ._llcrypt registered [ 790.015640] Key type .llcrypt registered [ 790.130513] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 791.637986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 791.661936] alg: No test for adler32 (adler32-zlib) [ 792.626902] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 792.805938] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 794.471593] Key type lgssc registered [ 795.916495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 803.646271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 803.693111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.130674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 809.727905] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 815.590966] LustreError: 29677:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 815.618912] LustreError: 29677:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 818.074730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.516201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 823.459584] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 830.861142] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 837.009725] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 850.938906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 851.235942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.690437] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 865.172501] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 874.811264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 884.192883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 884.202070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.225566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.219287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.220091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.763474] Lustre: server umount lustre-OST0000 complete [ 895.457090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 895.458907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 895.488985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.497809] Lustre: Skipped 2 previous similar messages [ 898.397485] Lustre: server umount lustre-MDT0000 complete [ 900.576895] LustreError: 29672:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.594661] LustreError: 29672:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 901.961819] LustreError: 29656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853786 with bad export cookie 5524596410775808416 [ 901.980135] LustreError: 29656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 901.985952] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 902.243764] Lustre: server umount lustre-MDT0001 complete [ 914.061438] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 917.159665] Key type lgssc unregistered [ 917.394237] LNet: 33581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.396827] LNetError: 33581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.413360] LNet: Removed LNI 192.168.201.117@tcp [ 918.244138] Key type .llcrypt unregistered [ 918.246076] Key type ._llcrypt unregistered [ 944.963750] Key type ._llcrypt registered [ 944.965572] Key type .llcrypt registered [ 945.129921] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 946.474890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 946.596075] alg: No test for adler32 (adler32-zlib) [ 947.868494] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 948.062745] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 949.775197] Key type lgssc registered [ 951.262051] Lustre: Echo OBD driver; http://www.lustre.org/ [ 959.642388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 959.668103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.191083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.754417] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 971.756865] LustreError: 34999:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.771886] LustreError: 34999:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 975.456269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.850561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 981.161246] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 988.636143] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 992.680256] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 998.694971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 999.053738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.690494] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1007.293251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1012.932663] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1021.920784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1021.940229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.978070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1022.945726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1022.949648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1022.951941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.967019] Lustre: Skipped 1 previous similar message [ 1027.342297] Lustre: server umount lustre-OST0000 complete [ 1032.160268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1032.170980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.180292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.045778] Lustre: server umount lustre-MDT0000 complete [ 1038.313254] LustreError: 35018:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.334293] LustreError: 35018:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1040.598217] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853924 with bad export cookie 5744809527993297426 [ 1040.599383] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.605308] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1040.984147] Lustre: server umount lustre-MDT0001 complete [ 1051.141603] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1054.179848] Key type lgssc unregistered [ 1054.542918] LNet: 38537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.555967] LNetError: 38537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.591915] LNet: Removed LNI 192.168.201.117@tcp [ 1056.439939] Key type .llcrypt unregistered [ 1056.442157] Key type ._llcrypt unregistered [ 1078.367792] Key type ._llcrypt registered [ 1078.369845] Key type .llcrypt registered [ 1078.457755] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1079.333882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1079.348638] alg: No test for adler32 (adler32-zlib) [ 1080.464562] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 1080.693444] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1082.383196] Key type lgssc registered [ 1083.476642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1090.501983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1090.517483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1091.879361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1096.395788] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1102.308341] LustreError: 39953:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.326280] LustreError: 39953:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1104.395127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1104.688251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1109.437740] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1116.938959] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1122.666315] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1129.815913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1130.128493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1136.884418] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1137.335809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1145.141864] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1155.047473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1155.047931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.054399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1155.079157] Lustre: Skipped 1 previous similar message [ 1156.660092] Lustre: server umount lustre-OST0000 complete [ 1176.545686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1176.555074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.573082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1176.581432] Lustre: Skipped 1 previous similar message [ 1180.642341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.642528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.668162] Lustre: Skipped 1 previous similar message [ 1182.888562] Lustre: server umount lustre-MDT0000 complete [ 1185.761434] LustreError: 39954:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.775570] LustreError: 39954:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1186.388615] LustreError: 39933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854070 with bad export cookie 6430864401448874261 [ 1186.395460] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.398875] LustreError: 39933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1192.800336] Lustre: server umount lustre-MDT0001 complete [ 1203.275964] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1206.436766] Key type lgssc unregistered [ 1206.739799] LNet: 43664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.745139] LNetError: 43664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.760280] LNet: Removed LNI 192.168.201.117@tcp [ 1207.677089] Key type .llcrypt unregistered [ 1207.687396] Key type ._llcrypt unregistered [ 1229.703218] Key type ._llcrypt registered [ 1229.705550] Key type .llcrypt registered [ 1229.833193] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1231.068495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1231.110784] alg: No test for adler32 (adler32-zlib) [ 1232.239293] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 1232.485134] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1234.175187] Key type lgssc registered [ 1235.711514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1243.757379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1243.788700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.395250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.916675] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1254.893043] LustreError: 45058:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.913833] LustreError: 45058:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1257.838689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.368350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1262.965628] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1269.506327] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1275.068838] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1279.956688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1280.253408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.305865] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1290.354892] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1290.494287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1296.099446] LustreError: 47635:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000006fce0aeb) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1296.180828] Lustre: server umount lustre-MDT0000 complete [ 1297.147458] LustreError: 46299:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.433066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1299.447490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.471409] LustreError: 46068:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.964340] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.042737] Lustre: Skipped 1 previous similar message [ 1302.173336] LustreError: 47548:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.205600] LustreError: 47548:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 1302.486845] LustreError: 45038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854186 with bad export cookie 10746138075604453149 [ 1302.511614] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1306.083137] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1306.093060] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1307.104583] LustreError: 45053:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1307.105686] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1307.139674] LustreError: 45053:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1308.811678] Lustre: server umount lustre-MDT0001 complete [ 1338.335186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1338.464661] Lustre: server umount lustre-OST0000 complete [ 1353.784888] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1356.993363] Key type lgssc unregistered [ 1357.226135] LNet: 48922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.232453] LNetError: 48922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.259331] LNet: Removed LNI 192.168.201.117@tcp [ 1358.186299] Key type .llcrypt unregistered [ 1358.190815] Key type ._llcrypt unregistered [ 1380.554245] Key type ._llcrypt registered [ 1380.556990] Key type .llcrypt registered [ 1380.651581] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1381.505369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1381.613201] alg: No test for adler32 (adler32-zlib) [ 1382.597492] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 1382.804839] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1384.495349] Key type lgssc registered [ 1385.384888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1393.028479] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1393.041649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1408.543547] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1413.664279] LustreError: 50290:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [ 1440.351808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1446.751864] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1454.833726] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1619.342578] Lustre: server umount lustre-OST0000 complete [ 1636.809454] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1640.054648] Key type lgssc unregistered [ 1640.394878] LNet: 51971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1640.402097] LNetError: 51971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.416776] LNet: Removed LNI 192.168.201.117@tcp [ 1641.226275] Key type .llcrypt unregistered [ 1641.228982] Key type ._llcrypt unregistered [ 1665.793620] Key type ._llcrypt registered [ 1665.795685] Key type .llcrypt registered [ 1665.888409] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1667.127161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1667.350535] alg: No test for adler32 (adler32-zlib) [ 1668.534221] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 1668.809770] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1670.471136] Key type lgssc registered [ 1671.665851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1679.837034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1679.871809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.590359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.589553] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1691.106556] LustreError: 53386:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1691.130988] LustreError: 53386:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1694.895906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.145736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1699.979969] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1707.745146] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1713.903736] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1720.586508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1720.759087] LustreError: 55207:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1721.159236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.289760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1727.867407] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1735.638847] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1741.796926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.800174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1741.811439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1741.811716] Lustre: Skipped 1 previous similar message [ 1741.828335] LustreError: Skipped 1 previous similar message [ 1746.912588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.919406] Lustre: Skipped 2 previous similar messages [ 1752.032490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1752.036122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 1752.048959] Lustre: Skipped 1 previous similar message [ 1752.206709] Lustre: server umount lustre-OST0000 complete [ 1756.642497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1756.658484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.669391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1761.921791] Lustre: server umount lustre-MDT0000 complete [ 1762.273176] LustreError: 53383:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.283878] LustreError: 53383:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1765.637697] LustreError: 53366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854649 with bad export cookie 10530832245486769421 [ 1765.655775] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1766.163517] Lustre: server umount lustre-MDT0001 complete [ 1779.119539] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1782.391815] Key type lgssc unregistered [ 1782.736137] LNet: 56930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.742332] LNetError: 56930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.760636] LNet: Removed LNI 192.168.201.117@tcp [ 1783.521642] Key type .llcrypt unregistered [ 1783.523869] Key type ._llcrypt unregistered [ 1806.399586] Key type ._llcrypt registered [ 1806.402622] Key type .llcrypt registered [ 1806.532327] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1808.033896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1808.103623] alg: No test for adler32 (adler32-zlib) [ 1809.165121] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 1809.428690] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1811.111239] Key type lgssc registered [ 1812.210988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1819.424785] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1819.449271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1834.976122] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1840.095393] LustreError: 58316:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [ 1866.783167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1873.659767] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1880.849479] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1886.518849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.099823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.138774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1891.605853] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1892.328107] LustreError: 58341:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.347579] LustreError: 58341:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1897.445057] LustreError: 58342:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1898.778179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.119528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1902.875245] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1910.531388] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1915.943449] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1917.922137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1917.925286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.962524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1923.046891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1923.063384] Lustre: Skipped 1 previous similar message [ 1928.163487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1928.172554] Lustre: Skipped 1 previous similar message [ 1932.255142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1932.442188] Lustre: server umount lustre-OST0000 complete [ 1955.296955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1955.309393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1955.327518] Lustre: Skipped 1 previous similar message [ 1955.345698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1955.360564] Lustre: Skipped 1 previous similar message [ 1958.442152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.266329] Lustre: server umount lustre-MDT0000 complete [ 1963.503323] LustreError: 59358:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1963.522811] LustreError: 59358:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1966.228497] LustreError: 58323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854850 with bad export cookie 2538718604693822704 [ 1966.235096] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1966.256440] LustreError: 58323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1972.743572] Lustre: server umount lustre-MDT0001 complete [ 1982.546860] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1985.499705] Key type lgssc unregistered [ 1985.826397] LNet: 62064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.841055] LNetError: 62064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.919090] LNet: Removed LNI 192.168.201.117@tcp [ 1987.916302] Key type .llcrypt unregistered [ 1987.921801] Key type ._llcrypt unregistered [ 2017.551389] Key type ._llcrypt registered [ 2017.556452] Key type .llcrypt registered [ 2017.806782] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2020.275167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2020.470614] alg: No test for adler32 (adler32-zlib) [ 2022.060123] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 2022.518828] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2024.319152] Key type lgssc registered [ 2026.064268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2036.288850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2036.312371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.081357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.834235] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2048.482744] LustreError: 63478:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.510668] LustreError: 63478:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2050.628810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.842406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2055.349504] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2062.440853] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2068.587792] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2074.541210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2074.854351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.433518] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2083.127730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2089.351552] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2108.898697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.901930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2108.908222] Lustre: Skipped 1 previous similar message [ 2108.919217] Lustre: Skipped 1 previous similar message [ 2109.238071] Lustre: server umount lustre-OST0000 complete [ 2114.018870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.041282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2114.051248] Lustre: Skipped 1 previous similar message [ 2119.138231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2119.141829] Lustre: Skipped 1 previous similar message [ 2119.817799] Lustre: server umount lustre-MDT0000 complete [ 2123.477382] LustreError: 63459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855007 with bad export cookie 17571563508508492969 [ 2123.492241] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2123.501587] LustreError: 63459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2124.001310] Lustre: server umount lustre-MDT0001 complete [ 2134.622417] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2137.774834] Key type lgssc unregistered [ 2138.037566] LNet: 67036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.042908] LNetError: 67036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.057771] LNet: Removed LNI 192.168.201.117@tcp [ 2138.744680] Key type .llcrypt unregistered [ 2138.747113] Key type ._llcrypt unregistered [ 2160.330873] Key type ._llcrypt registered [ 2160.333450] Key type .llcrypt registered [ 2160.416252] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2161.265352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2161.322116] alg: No test for adler32 (adler32-zlib) [ 2162.427822] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 2162.735796] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2164.431169] Key type lgssc registered [ 2165.621404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2173.578920] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 14:24:17 (1787855057) [ 2180.232367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2180.243515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.806029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2186.391694] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2191.351952] LustreError: 68719:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.372579] LustreError: 68719:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2195.105773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.417614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2200.029342] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2207.096686] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2213.123515] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2220.173430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2220.546437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.762329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2227.140174] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2234.917561] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2243.624554] Lustre: Failing over lustre-MDT0000 [ 2243.979103] Lustre: server umount lustre-MDT0000 complete [ 2245.609856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.634348] Lustre: Skipped 2 previous similar messages [ 2245.635104] LustreError: 68715:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.649430] LustreError: 68715:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2246.875131] LustreError: 68715:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.723641] LustreError: 68720:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.731417] LustreError: 68720:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2253.019957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.242840] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2253.404672] Lustre: *** cfs_fail_loc=135, val=0*** [ 2253.508951] LustreError: 71807:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2253.517032] LustreError: 71807:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.117@tcp: cfg command failed: rc = -2 [ 2253.526044] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2253.526044] [ 2253.535595] LustreError: MGC192.168.201.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2253.558037] LustreError: 71796:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2253.573819] LustreError: 71796:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2253.578850] LustreError: 71796:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2253.581420] LustreError: 71796:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2253.614410] Lustre: server umount lustre-MDT0000 complete [ 2253.621448] LustreError: 71796:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2255.844091] LustreError: 68714:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.871663] LustreError: 68714:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2260.539987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.668550] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.913492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.994747] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2262.192281] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2265.929936] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2266.095378] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2266.127633] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2266.180774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2276.322903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2276.337844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2276.341199] Lustre: Skipped 1 previous similar message [ 2279.822469] Lustre: server umount lustre-OST0000 complete [ 2286.561188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2286.565228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.578336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.613355] Lustre: Skipped 1 previous similar message [ 2286.624695] Lustre: Skipped 2 previous similar messages [ 2289.383989] Lustre: server umount lustre-MDT0000 complete [ 2291.680431] LustreError: 68719:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2291.710812] LustreError: 68719:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2292.954651] LustreError: 68699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855177 with bad export cookie 14532939638606984887 [ 2292.966209] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2293.570241] Lustre: server umount lustre-MDT0001 complete [ 2309.308031] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2312.282610] Key type lgssc unregistered [ 2312.628478] LNet: 73586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.632155] LNetError: 73586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2313.705282] LNet: Removed LNI 192.168.201.117@tcp [ 2315.121236] Key type .llcrypt unregistered [ 2315.134831] Key type ._llcrypt unregistered [ 2342.330533] Key type ._llcrypt registered [ 2342.331819] Key type .llcrypt registered [ 2342.410982] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2343.266301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2343.413142] alg: No test for adler32 (adler32-zlib) [ 2344.606748] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 2344.859855] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2346.527665] Key type lgssc registered [ 2347.633192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2355.604376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2355.627749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.227797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.835118] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2367.469941] LustreError: 75001:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2367.508412] LustreError: 75001:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2371.855067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.431978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2380.132900] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2391.058371] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2398.385523] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2404.622248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2405.008716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.095521] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2420.860770] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2427.586794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2437.486591] Lustre: Failing over lustre-MDT0000 [ 2437.855696] LustreError: 74996:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.105205] Lustre: server umount lustre-MDT0000 complete [ 2438.121834] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.147618] Lustre: Skipped 2 previous similar messages [ 2442.904479] LustreError: 76025:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.925791] LustreError: 76025:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2448.019324] LustreError: 76025:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.045230] LustreError: 76025:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2448.096097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.196467] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.350899] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2448.362242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2448.501039] LustreError: 78111:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2448.505219] LustreError: 78111:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.117@tcp: cfg command failed: rc = -9 [ 2448.518151] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2448.518151] [ 2448.529102] LustreError: MGC192.168.201.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2448.542550] LustreError: 78099:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2448.563170] LustreError: 78099:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2448.568314] LustreError: 78099:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2448.574818] LustreError: 78099:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2448.608929] Lustre: server umount lustre-MDT0000 complete [ 2448.611684] LustreError: 78099:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2453.160732] LustreError: 74996:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2455.559807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2455.650358] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2455.660555] LustreError: 74991:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2455.902659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2455.952942] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2458.256818] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2460.915938] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2461.175806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2461.272509] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2461.353137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2471.398972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.414350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2471.418236] Lustre: Skipped 1 previous similar message [ 2471.439866] Lustre: Skipped 3 previous similar messages [ 2476.047917] Lustre: server umount lustre-OST0000 complete [ 2481.635909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.658671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2481.676974] Lustre: Skipped 1 previous similar message [ 2481.706282] Lustre: Skipped 1 previous similar message [ 2486.754182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2486.766452] Lustre: Skipped 1 previous similar message [ 2487.011794] Lustre: server umount lustre-MDT0000 complete [ 2490.850130] LustreError: 74995:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.880066] LustreError: 74995:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 2491.363390] LustreError: 74981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855375 with bad export cookie 17986761634936701212 [ 2491.371816] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2491.751531] Lustre: server umount lustre-MDT0001 complete [ 2503.442771] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2506.385421] Key type lgssc unregistered [ 2506.818807] LNet: 79894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2506.836348] LNetError: 79894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2506.869848] LNet: Removed LNI 192.168.201.117@tcp [ 2507.820951] Key type .llcrypt unregistered [ 2507.823934] Key type ._llcrypt unregistered [ 2524.065816] Key type ._llcrypt registered [ 2524.070704] Key type .llcrypt registered [ 2524.204806] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2524.471444] Key type .llcrypt unregistered [ 2524.473159] Key type ._llcrypt unregistered [ 2553.971132] Key type ._llcrypt registered [ 2553.973965] Key type .llcrypt registered [ 2554.102416] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2555.678442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2555.701409] alg: No test for adler32 (adler32-zlib) [ 2556.817905] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 2557.182174] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2558.992473] Key type lgssc registered [ 2560.963154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2572.752711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2572.798288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.644338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.756158] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2584.054200] LustreError: 81810:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.070210] LustreError: 81810:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2589.161724] LustreError: 81811:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.914038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.210763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2595.971541] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2602.967356] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2608.416965] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2617.504849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2617.880435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.938779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2626.412923] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2637.791853] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2649.568456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2649.581691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2649.604143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2651.618393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2651.621808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2653.305993] Lustre: server umount lustre-OST0000 complete [ 2657.761122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2657.771395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.803665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.806113] Lustre: Skipped 1 previous similar message [ 2661.858081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2661.867127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.319722] Lustre: server umount lustre-MDT0000 complete [ 2666.976777] LustreError: 84102:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.011686] LustreError: 84102:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2667.538931] LustreError: 81792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855551 with bad export cookie 7330723846782781826 [ 2667.540318] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.547785] LustreError: 81792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2668.079972] Lustre: server umount lustre-MDT0001 complete [ 2681.226186] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2684.023608] Key type lgssc unregistered [ 2684.307083] LNet: 85351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2684.316525] LNetError: 85351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2684.326182] LNet: Removed LNI 192.168.201.117@tcp [ 2685.296964] Key type .llcrypt unregistered [ 2685.300297] Key type ._llcrypt unregistered [ 2711.436978] Key type ._llcrypt registered [ 2711.439293] Key type .llcrypt registered [ 2711.609971] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2712.959647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2712.994672] alg: No test for adler32 (adler32-zlib) [ 2714.311418] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 2714.783160] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2716.543226] Key type lgssc registered [ 2717.615376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2726.684835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2726.707624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.043293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.047792] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2738.659327] LustreError: 86764:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.683349] LustreError: 86764:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2743.784312] LustreError: 86763:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.187740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.537482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2749.367523] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2759.122983] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2765.212375] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2772.612985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.083078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2774.139964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2779.601816] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2788.021049] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2800.096304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2800.106761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.109470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2800.117550] Lustre: Skipped 1 previous similar message [ 2803.537267] Lustre: server umount lustre-OST0000 complete [ 2810.336681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.350457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2810.352864] Lustre: Skipped 2 previous similar messages [ 2813.579861] Lustre: server umount lustre-MDT0000 complete [ 2815.456218] LustreError: 86763:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.480992] LustreError: 86763:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2818.011855] LustreError: 86744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855702 with bad export cookie 6950468629870978795 [ 2818.014092] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2818.020659] LustreError: 86744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2818.538831] Lustre: server umount lustre-MDT0001 complete [ 2832.189954] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2835.664313] Key type lgssc unregistered [ 2835.990742] LNet: 90301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2836.001811] LNetError: 90301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2837.029556] LNet: Removed LNI 192.168.201.117@tcp [ 2838.322160] Key type .llcrypt unregistered [ 2838.330547] Key type ._llcrypt unregistered [ 2863.694177] Key type ._llcrypt registered [ 2863.702319] Key type .llcrypt registered [ 2863.914245] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2865.289855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2865.466235] alg: No test for adler32 (adler32-zlib) [ 2866.596518] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 2866.937639] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2868.663305] Key type lgssc registered [ 2869.931524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2878.564435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2878.583718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.274095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.992832] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2890.724501] LustreError: 91715:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.756288] LustreError: 91715:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2894.831743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.244666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2900.547048] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2908.978840] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2915.680798] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2924.146759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2924.540679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2925.596453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2931.668245] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2939.789937] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2955.234308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2955.236818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2955.239545] Lustre: Skipped 1 previous similar message [ 2955.242820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2958.351249] Lustre: server umount lustre-OST0000 complete [ 2965.473348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.480392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2965.485192] Lustre: Skipped 1 previous similar message [ 2969.364215] Lustre: server umount lustre-MDT0000 complete [ 2970.593715] LustreError: 91711:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.250741] LustreError: 91697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855857 with bad export cookie 2870657173985675326 [ 2973.286251] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2974.115135] Lustre: server umount lustre-MDT0001 complete [ 2988.691365] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2992.295223] Key type lgssc unregistered [ 2992.675576] LNet: 95267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2992.693744] LNetError: 95267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2993.765808] LNet: Removed LNI 192.168.201.117@tcp [ 2994.766477] Key type .llcrypt unregistered [ 2994.768204] Key type ._llcrypt unregistered [ 3018.859667] Key type ._llcrypt registered [ 3018.861197] Key type .llcrypt registered [ 3018.931688] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3020.050440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3020.065251] alg: No test for adler32 (adler32-zlib) [ 3021.166234] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 3021.444305] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 3023.119152] Key type lgssc registered [ 3024.674167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3035.755994] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3035.769793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3051.490877] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3056.609532] LustreError: 96655:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [ 3083.232547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.742929] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3099.211959] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3106.248257] Lustre: server umount lustre-OST0000 complete [ 3114.652576] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 14:39:57 (1787855997) [ 3121.878790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.171654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3127.272926] LustreError: 98296:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.289490] LustreError: 98296:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3128.166360] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3132.385520] LustreError: 98297:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.458735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3137.505447] LustreError: 98296:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.840960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3143.295896] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3150.973378] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3156.453933] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3164.047797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3164.546553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.605688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3171.627284] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3180.198362] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3198.434577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3198.442149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.442859] LustreError: Skipped 1 previous similar message [ 3198.457475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3198.459893] Lustre: Skipped 1 previous similar message [ 3198.474943] Lustre: Skipped 1 previous similar message [ 3201.658173] Lustre: server umount lustre-OST0000 complete [ 3208.675565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3208.690760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3208.694259] Lustre: Skipped 1 previous similar message [ 3212.478396] Lustre: server umount lustre-MDT0000 complete [ 3213.795058] LustreError: 100527:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.815379] LustreError: 100527:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3216.714181] LustreError: 100139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856100 with bad export cookie 8377074675223884946 [ 3216.732520] LustreError: 100139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3216.736312] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3217.090405] Lustre: server umount lustre-MDT0001 complete [ 3229.883220] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 3233.444242] Key type lgssc unregistered [ 3233.856531] LNet: 101992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3233.860527] LNetError: 101992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3233.876176] LNet: Removed LNI 192.168.201.117@tcp [ 3235.115765] Key type .llcrypt unregistered [ 3235.117756] Key type ._llcrypt unregistered [ 3283.740434] Key type ._llcrypt registered [ 3283.749246] Key type .llcrypt registered [ 3283.941527] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3285.146981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3285.399222] alg: No test for adler32 (adler32-zlib) [ 3286.666574] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 3287.087106] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 3288.791193] Key type lgssc registered [ 3290.337156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3299.928680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3299.952237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.365230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.390836] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3311.588091] LustreError: 104177:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3311.625618] LustreError: 104177:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3315.784462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3316.233401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3322.097062] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3330.220504] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3336.465516] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3343.846849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3344.236389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3346.306216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3351.020721] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3359.301252] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3372.001683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.016083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3372.026895] Lustre: Skipped 1 previous similar message [ 3372.029691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3375.676821] Lustre: server umount lustre-OST0000 complete [ 3382.243684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.258919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.261374] Lustre: Skipped 2 previous similar messages [ 3382.292269] Lustre: Skipped 1 previous similar message [ 3385.992891] Lustre: server umount lustre-MDT0000 complete [ 3387.361145] LustreError: 104177:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.377735] LustreError: 104177:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3389.896470] LustreError: 104158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856274 with bad export cookie 11231854001471472861 [ 3389.903319] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3389.913040] LustreError: 104158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3390.449041] Lustre: server umount lustre-MDT0001 complete [ 3403.123650] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 3406.354765] Key type lgssc unregistered [ 3406.660240] LNet: 107716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3406.666768] LNetError: 107716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3406.688975] LNet: Removed LNI 192.168.201.117@tcp [ 3407.770154] Key type .llcrypt unregistered [ 3407.771423] Key type ._llcrypt unregistered [ 3428.546172] Key type ._llcrypt registered [ 3428.553777] Key type .llcrypt registered [ 3428.729471] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3430.362555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3430.537900] alg: No test for adler32 (adler32-zlib) [ 3431.651206] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 3431.899361] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 3433.591114] Key type lgssc registered [ 3434.909375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3443.198065] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3443.223451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3458.783459] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3463.903348] LustreError: 108964:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [ 3490.527188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.138445] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3506.987275] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3515.172516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3515.529355] LustreError: MGC192.168.201.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3515.548691] LustreError: 109988:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3515.562835] LustreError: 109988:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3515.583531] LustreError: 109988:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3515.676834] Lustre: server umount lustre-MDT0000 complete [ 3515.680357] LustreError: 109988:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3533.793358] Lustre: 108300:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856402/real 1787856402] req@00000000bbc1daf3 x1874703224409472/t0(0) o400->MGC192.168.201.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787856418 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3534.422644] Lustre: server umount lustre-OST0000 complete [ 3544.826971] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 3554.939210] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3593.699978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.914848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3593.934672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3593.999557] Lustre: lustre-MDT0000: new disk, initializing [ 3594.099579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3594.125671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3599.380822] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3612.241819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3612.428571] Lustre: 112780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3612.529276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3612.532436] Lustre: Skipped 1 previous similar message [ 3612.597696] Lustre: lustre-MDT0001: new disk, initializing [ 3612.710969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3612.752172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3612.766017] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3618.159642] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3630.142526] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3638.711473] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3647.211349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3647.534409] Lustre: lustre-OST0000: new disk, initializing [ 3647.536933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3647.540816] Lustre: 114749:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3647.619696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3654.435610] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3654.773373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3654.786834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3654.897520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3666.531986] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3672.030392] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3672.280892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3678.425177] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3678.614352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3685.860506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3685.878259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.906157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3685.917621] Lustre: Skipped 1 previous similar message [ 3687.398456] Lustre: server umount lustre-OST0000 complete [ 3694.568719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3694.577268] LustreError: Skipped 1 previous similar message [ 3694.580130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.609937] Lustre: Skipped 1 previous similar message [ 3694.617568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3696.096322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3696.105792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.228794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3701.231512] Lustre: Skipped 2 previous similar messages [ 3706.335181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3706.338889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3706.363988] Lustre: Skipped 1 previous similar message [ 3706.493577] Lustre: server umount lustre-MDT0000 complete [ 3709.538170] LustreError: 112771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856593 with bad export cookie 9606996498125480393 [ 3709.547808] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3709.557722] LustreError: 112771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3709.809746] Lustre: server umount lustre-MDT0001 complete [ 3719.250866] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 14:50:02 (1787856602) [ 3723.627664] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3742.930028] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 3754.705983] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3790.284368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.572926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3790.594515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3790.699492] Lustre: lustre-MDT0000: new disk, initializing [ 3790.841139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.884898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3795.981435] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3808.232035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3808.386193] Lustre: 119655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3808.468828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3808.473352] Lustre: Skipped 1 previous similar message [ 3808.542517] Lustre: lustre-MDT0001: new disk, initializing [ 3808.647382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3808.690255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3808.708666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3813.531605] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3823.376453] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3830.046589] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3837.683092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3838.007760] Lustre: lustre-OST0000: new disk, initializing [ 3838.017132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3838.022966] Lustre: 121624:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3838.088745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.211062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3845.224111] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3845.250964] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3845.347297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3855.500363] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3861.072993] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3861.315109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3867.333538] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3867.682198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3871.208745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3871.220057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.224286] LustreError: Skipped 1 previous similar message [ 3871.226273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3871.245855] Lustre: Skipped 1 previous similar message [ 3871.278666] Lustre: Skipped 1 previous similar message [ 3876.405669] Lustre: server umount lustre-OST0000 complete [ 3880.425863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3880.432784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.770807] Lustre: server umount lustre-MDT0000 complete [ 3890.956223] LustreError: 119646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856775 with bad export cookie 9606996498125481275 [ 3890.964831] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3890.968771] LustreError: 119646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3891.588961] Lustre: server umount lustre-MDT0001 complete [ 3900.375664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3901.045917] LustreError: 123840:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3901.072876] LustreError: 123840:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 3901.112345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3906.486327] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3906.528662] LustreError: 123841:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3911.655018] LustreError: 123840:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3915.257334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3920.923537] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3928.347402] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3934.625937] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3941.855238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3949.535875] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3958.308975] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3968.291844] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3976.164649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3976.165186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3976.181811] Lustre: Skipped 1 previous similar message [ 3976.195641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3976.203677] Lustre: Skipped 4 previous similar messages [ 3979.058800] Lustre: server umount lustre-OST0000 complete [ 3997.151385] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3997.359559] Lustre: server umount lustre-MDT0000 complete [ 4001.693300] LustreError: 125683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856885 with bad export cookie 9606996498125482157 [ 4001.700785] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4001.730503] LustreError: 125683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4002.094899] Lustre: server umount lustre-MDT0001 complete [ 4015.248350] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4019.270148] Key type lgssc unregistered [ 4019.700577] LNet: 127476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4019.707894] LNetError: 127476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4020.777463] LNet: Removed LNI 192.168.201.117@tcp [ 4021.661227] Key type .llcrypt unregistered [ 4021.662901] Key type ._llcrypt unregistered [ 4042.693960] Key type ._llcrypt registered [ 4042.697840] Key type .llcrypt registered [ 4042.845241] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 4058.491832] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4059.724156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4059.865754] alg: No test for adler32 (adler32-zlib) [ 4060.923336] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 4061.175112] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 4062.855304] Key type lgssc registered [ 4064.142378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4099.158110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4099.221516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.491876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4100.537264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4100.627455] Lustre: lustre-MDT0000: new disk, initializing [ 4100.716687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4100.726319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4104.946340] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4115.258355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.418768] Lustre: 130633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4115.492525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4115.496146] Lustre: Skipped 1 previous similar message [ 4115.597036] Lustre: lustre-MDT0001: new disk, initializing [ 4115.705695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4115.784551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4115.803682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4120.140205] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4130.166239] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4136.021540] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4142.449213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4142.870499] Lustre: lustre-OST0000: new disk, initializing [ 4142.878225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4142.889946] Lustre: 132601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4143.006550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4149.241624] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4152.425851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4152.434155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4152.521912] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4158.017728] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4162.754466] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4162.919520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4167.328962] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4167.548133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4173.283663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4173.284719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4173.297508] LustreError: Skipped 1 previous similar message [ 4173.299667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4173.337491] Lustre: Skipped 1 previous similar message [ 4175.334547] Lustre: server umount lustre-OST0000 complete [ 4182.496906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4182.502411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4182.518053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4182.521022] Lustre: Skipped 1 previous similar message [ 4183.522930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4183.524627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4183.555687] Lustre: Skipped 1 previous similar message [ 4186.657465] Lustre: server umount lustre-MDT0000 complete [ 4191.388492] LustreError: 130624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857075 with bad export cookie 9855895405802423150 [ 4191.398684] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4191.411026] LustreError: 130624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4191.816717] Lustre: server umount lustre-MDT0001 complete [ 4200.506957] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 14:58:03 (1787857083) [ 4206.781228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.457794] LustreError: 135013:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4207.475198] LustreError: 135013:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4207.601811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.324495] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4212.719599] LustreError: 135014:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.806167] LustreError: 135013:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.013028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.541154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4227.129427] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4235.133642] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4240.606598] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4246.690275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4246.696133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4246.725326] Lustre: Skipped 1 previous similar message [ 4249.489698] Lustre: server umount lustre-MDT0000 complete [ 4252.133955] LustreError: 135008:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4252.166041] LustreError: 135008:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4254.427861] LustreError: 134993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857138 with bad export cookie 9855895405802424032 [ 4254.430393] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4254.434458] LustreError: 134993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4255.091703] Lustre: server umount lustre-MDT0001 complete [ 4265.087379] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 14:59:08 (1787857148) [ 4271.590209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4287.199636] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4292.319558] LustreError: 137429:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [ 4318.052219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.201715] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4334.487357] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4336.941875] Lustre: server umount lustre-OST0000 complete [ 4346.494938] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 15:00:29 (1787857229) [ 4353.942513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4354.601136] LustreError: 138876:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.701759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4359.907808] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4364.262319] LustreError: 138877:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.276097] LustreError: 138877:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4368.944991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4374.669212] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4383.424579] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4389.722994] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4397.239348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4397.783479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.792722] Lustre: Skipped 1 previous similar message [ 4405.076978] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4413.112821] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4424.673157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4424.692663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4424.693781] Lustre: Skipped 2 previous similar messages [ 4424.706500] Lustre: Skipped 3 previous similar messages [ 4428.040485] Lustre: server umount lustre-MDT0000 complete [ 4429.797152] LustreError: 139841:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4429.833063] LustreError: 139841:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4432.272648] LustreError: 138857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857316 with bad export cookie 9855895405802424697 [ 4432.282228] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4432.640139] Lustre: server umount lustre-MDT0001 complete [ 4446.596472] Lustre: server umount lustre-OST0000 complete [ 4455.961982] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:02:18 (1787857338) [ 4463.850624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.375458] LustreError: 142543:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.478236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4469.731439] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4479.094325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.637115] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4493.805744] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4500.495711] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4507.794208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4509.309980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4516.540307] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4524.529806] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4529.339736] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4529.533927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4534.752827] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4534.991818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4540.386339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4540.389578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4540.418916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4540.435344] Lustre: Skipped 1 previous similar message [ 4543.203235] Lustre: server umount lustre-OST0000 complete [ 4559.842495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4559.853553] Lustre: Skipped 4 previous similar messages [ 4561.375729] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4561.843434] Lustre: server umount lustre-MDT0000 complete [ 4564.964654] LustreError: 142540:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4564.981286] LustreError: 142540:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 4566.244229] LustreError: 142523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857450 with bad export cookie 9855895405802425768 [ 4566.245338] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4566.256322] LustreError: 142523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4566.869522] Lustre: server umount lustre-MDT0001 complete [ 4575.705754] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:04:18 (1787857458) [ 4583.737169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4599.392343] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4604.512455] LustreError: 146704:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [ 4630.239695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4630.255535] Lustre: Skipped 2 previous similar messages [ 4637.030674] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4645.038357] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4652.316481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4652.990108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4658.513554] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4667.501376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4673.107884] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4681.472724] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4687.512779] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4693.082352] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4693.290338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4699.402326] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4699.723231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4703.715363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4703.716523] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4703.733469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4703.761603] Lustre: Skipped 4 previous similar messages [ 4708.520918] Lustre: server umount lustre-MDT0000 complete [ 4708.867514] LustreError: 147740:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.887638] LustreError: 147740:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 4712.706435] LustreError: 146710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857596 with bad export cookie 9855895405802426790 [ 4712.717446] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4713.026262] Lustre: server umount lustre-MDT0001 complete [ 4727.917467] Lustre: server umount lustre-OST0000 complete [ 4737.111461] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 15:07:00 (1787857620) [ 4744.033802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4759.647949] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4764.768854] LustreError: 150904:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [ 4798.961988] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4807.664142] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4815.866844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.510436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4821.553405] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4831.183219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4836.446449] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4845.155704] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4851.539238] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4859.090047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4859.480583] Lustre: lustre-OST0001: new disk, initializing [ 4859.487367] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4859.512473] Lustre: 153777:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4865.629165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4865.637288] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4865.931693] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4867.576704] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4878.853491] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4884.604772] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4884.960288] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4888.933764] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4889.173745] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4891.621911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.634355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4891.645778] Lustre: Skipped 2 previous similar messages [ 4891.669047] Lustre: Skipped 3 previous similar messages [ 4897.505293] Lustre: server umount lustre-OST0000 complete [ 4906.981349] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4908.360372] Lustre: server umount lustre-OST0001 complete [ 4921.108134] Lustre: server umount lustre-MDT0000 complete [ 4926.430565] LustreError: 153776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857810 with bad export cookie 9855895405802427742 [ 4926.449725] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4926.463701] LustreError: 153776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4927.516847] Lustre: server umount lustre-MDT0001 complete [ 4956.196843] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 15:10:39 (1787857839) [ 4957.926206] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4959.843983] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4961.540764] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:10:45 (1787857845) [ 4969.060887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4969.292397] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4969.316214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4969.480573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4969.489163] Lustre: Skipped 6 previous similar messages [ 4974.911692] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4984.272938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4984.332780] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4984.356470] Lustre: 157339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4989.311123] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4996.161139] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5001.246736] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5007.967378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5008.254301] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5015.106874] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5018.227530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5022.473915] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5027.539836] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5027.747704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5033.431936] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5033.846903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5038.560253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5038.564380] LustreError: Skipped 1 previous similar message [ 5038.566737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.579451] Lustre: Skipped 4 previous similar messages [ 5038.588633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5038.593561] Lustre: Skipped 6 previous similar messages [ 5042.845239] Lustre: server umount lustre-OST0000 complete [ 5066.632843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5073.891512] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5079.276538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.298403] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5082.565217] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5088.970670] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5089.232354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5089.573603] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.590432] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5093.928046] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5094.115134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5099.629958] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d6f10a2e000.ost_server_uuid 50 [ 5101.199415] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d6f10a2e000.ost_server_uuid in FULL state after 0 sec [ 5101.270557] LustreError: 157347:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.17@tcp) returned error from blocking AST (req@0000000015b477fc x1874703884226816 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000004df04bd5/0x88c72ca71e421eb8 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.17@tcp remote: 0xab8b4ac288aa8df6 expref: 5 pid: 157346 timeout: 5201 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5101.290853] LustreError: lustre-MDT0000: A client on nid 192.168.201.17@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5101.295863] LustreError: 157335:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.17@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000004df04bd5/0x88c72ca71e421eb8 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.17@tcp remote: 0xab8b4ac288aa8df6 expref: 6 pid: 157346 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5107.183717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5107.197142] LustreError: Skipped 1 previous similar message [ 5112.117327] Lustre: server umount lustre-OST0000 complete [ 5122.755855] Lustre: server umount lustre-MDT0000 complete [ 5123.558757] LustreError: 157350:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5123.590531] LustreError: 157350:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 14 previous similar messages [ 5127.067712] LustreError: 157330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858011 with bad export cookie 9855895405802428925 [ 5127.072082] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5127.074367] LustreError: 157330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5133.912635] Lustre: server umount lustre-MDT0001 complete [ 5144.749242] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5148.576001] Key type lgssc unregistered [ 5149.072871] LNet: 163458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5149.077444] LNetError: 163458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5149.111040] LNet: Removed LNI 192.168.201.117@tcp [ 5150.580215] Key type .llcrypt unregistered [ 5150.585639] Key type ._llcrypt unregistered [ 5177.479374] Key type ._llcrypt registered [ 5177.483973] Key type .llcrypt registered [ 5177.592119] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5179.032979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5179.048210] alg: No test for adler32 (adler32-zlib) [ 5180.079248] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 5180.314642] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5181.967474] Key type lgssc registered [ 5183.432847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5192.945086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5192.966911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5194.475706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5199.030967] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5203.952575] LustreError: 164872:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5203.974582] LustreError: 164872:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5208.309958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5208.668059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5212.819069] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5221.795365] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5227.558446] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5234.751656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5235.239884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5239.355576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5242.518766] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5250.639351] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5259.216910] Lustre: Failing over lustre-MDT0000 [ 5259.518076] Lustre: server umount lustre-MDT0000 complete [ 5259.745201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5259.755070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5259.771796] LustreError: 164872:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5263.840729] LustreError: 165885:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5263.846366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.860601] Lustre: Skipped 1 previous similar message [ 5268.968687] LustreError: 165885:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5268.995717] LustreError: 165885:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 5271.788406] LustreError: 164851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.17@tcp arrived at 1787858156 with bad export cookie 12446921581065315938 [ 5274.098071] LustreError: 164872:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.124308] LustreError: 164872:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5280.223150] Lustre: 164182:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858148/real 1787858148] req@0000000076e648e1 x1874705057344512/t0(0) o400->MGC192.168.201.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787858164 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5280.256861] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5280.947806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5285.348404] LustreError: 164866:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.396211] LustreError: 164866:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 8 previous similar messages [ 5290.865487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5290.935992] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5296.096953] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5296.103651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5296.400869] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5298.584379] Lustre: lustre-MDT0000: Denying connection for new client 61784533-d714-409a-94fd-000c578cce01 (at 192.168.201.17@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5309.928916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5309.937262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5309.954073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5311.459098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5311.481023] Lustre: Skipped 1 previous similar message [ 5313.404983] Lustre: server umount lustre-OST0000 complete [ 5317.677100] LustreError: 168606:0:(ldlm_lib.c:3039:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5317.695696] Lustre: 167953:0:(ldlm_lib.c:2439:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5317.703687] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5317.720082] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5317.734804] LustreError: 167953:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002f7bb63a x1874705057365248/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5317.762520] LustreError: 167953:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5317.772659] LustreError: 167953:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5317.782747] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5318.109341] Lustre: server umount lustre-MDT0000 complete [ 5321.704588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.719323] Lustre: Skipped 1 previous similar message [ 5321.743879] LustreError: 164866:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5321.785311] LustreError: 164866:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 5321.967493] LustreError: 164853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858206 with bad export cookie 12446921581065316043 [ 5321.976750] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5321.979435] LustreError: 164853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5321.992455] LustreError: 164180:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5324.406836] Lustre: server umount lustre-MDT0001 complete [ 5335.933794] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5338.849950] Key type lgssc unregistered [ 5339.139277] LNet: 169363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5339.162345] LNetError: 169363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5339.187232] LNet: Removed LNI 192.168.201.117@tcp [ 5340.260317] Key type .llcrypt unregistered [ 5340.262162] Key type ._llcrypt unregistered [ 5365.375513] Key type ._llcrypt registered [ 5365.377897] Key type .llcrypt registered [ 5365.505258] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5366.814271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5367.102607] alg: No test for adler32 (adler32-zlib) [ 5368.330681] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 5368.649493] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5370.407853] Key type lgssc registered [ 5371.535832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5379.790985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5379.847911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5381.426165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.888535] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5390.823863] LustreError: 170776:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5390.832534] LustreError: 170776:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5395.293790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5395.924263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5400.014742] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5407.168093] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5413.475211] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5420.931871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5421.368623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5426.511711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5427.074328] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5433.933620] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5442.024940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.044667] Lustre: Skipped 1 previous similar message [ 5442.047860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5443.683844] Lustre: server umount lustre-OST0000 complete [ 5452.259329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5452.269461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.272366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5452.272374] Lustre: Skipped 1 previous similar message [ 5452.291479] Lustre: Skipped 1 previous similar message [ 5453.613853] Lustre: server umount lustre-MDT0000 complete [ 5457.147241] LustreError: 170759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858341 with bad export cookie 12332058091883383295 [ 5457.154787] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5457.161447] LustreError: 170759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5457.591871] Lustre: server umount lustre-MDT0001 complete [ 5469.046825] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5472.673530] Key type lgssc unregistered [ 5473.009723] LNet: 174310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5473.025406] LNetError: 174310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5473.050774] LNet: Removed LNI 192.168.201.117@tcp [ 5474.131354] Key type .llcrypt unregistered [ 5474.135805] Key type ._llcrypt unregistered [ 5509.260201] Key type ._llcrypt registered [ 5509.261950] Key type .llcrypt registered [ 5509.343742] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5510.064570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5510.128829] alg: No test for adler32 (adler32-zlib) [ 5511.555451] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 5511.915595] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5513.703392] Key type lgssc registered [ 5515.209566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5525.773669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5525.796861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.524247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5534.055952] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5536.817645] LustreError: 176014:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5536.836082] LustreError: 176014:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5541.867422] LustreError: 176015:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5544.874224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5545.299416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5550.987879] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5559.787359] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5566.719303] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5575.310724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5575.904580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.090639] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5595.074070] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5597.504406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5607.906545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5607.933874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.958269] Lustre: Skipped 1 previous similar message [ 5607.961552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5607.980440] Lustre: Skipped 1 previous similar message [ 5613.031394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5613.035569] Lustre: Skipped 1 previous similar message [ 5614.184237] Lustre: server umount lustre-OST0000 complete [ 5618.149519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.182590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5618.191494] Lustre: Skipped 1 previous similar message [ 5623.267066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5623.280466] Lustre: Skipped 1 previous similar message [ 5624.269826] Lustre: server umount lustre-MDT0000 complete [ 5628.036071] LustreError: 175996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858512 with bad export cookie 48187538659299533 [ 5628.054136] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5628.532696] Lustre: server umount lustre-MDT0001 complete [ 5639.547491] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5642.147549] Key type lgssc unregistered [ 5642.407748] LNet: 179574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5642.411850] LNetError: 179574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5642.425955] LNet: Removed LNI 192.168.201.117@tcp [ 5643.255432] Key type .llcrypt unregistered [ 5643.258165] Key type ._llcrypt unregistered [ 5665.600797] Key type ._llcrypt registered [ 5665.604219] Key type .llcrypt registered [ 5665.704752] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5666.514123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5666.684704] alg: No test for adler32 (adler32-zlib) [ 5667.833392] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 5668.070148] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5669.743183] Key type lgssc registered [ 5671.018694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5682.670253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5682.687881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5684.050210] Lustre: *** cfs_fail_loc=135, val=0*** [ 5684.138173] LustreError: 181100:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5684.144122] LustreError: 181100:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.117@tcp: cfg command failed: rc = -2 [ 5684.155631] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5684.155631] [ 5684.164405] LustreError: MGC192.168.201.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5684.174712] LustreError: 181061:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5684.184705] LustreError: 181061:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5684.189224] LustreError: 181061:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5684.197344] LustreError: 181061:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5684.264936] Lustre: server umount lustre-MDT0000 complete [ 5684.266409] LustreError: 181061:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5691.387700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.916147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5696.566694] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5702.123351] LustreError: 181471:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5702.151128] LustreError: 181471:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5703.143622] LustreError: 181470:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5705.782194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5706.159761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5710.947542] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5718.601948] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5724.847122] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5728.750834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.759404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5728.782047] Lustre: Skipped 1 previous similar message [ 5730.850956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5730.858154] Lustre: Skipped 2 previous similar messages [ 5733.209123] Lustre: server umount lustre-MDT0000 complete [ 5735.906209] LustreError: 182199:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5735.937212] LustreError: 182199:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5737.539238] LustreError: 181453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858621 with bad export cookie 6859184280430523699 [ 5737.551663] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5737.556592] LustreError: 181453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5738.253471] Lustre: server umount lustre-MDT0001 complete [ 5751.352645] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5755.854918] Key type lgssc unregistered [ 5756.116889] LNet: 183795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5756.122088] LNetError: 183795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5756.149194] LNet: Removed LNI 192.168.201.117@tcp [ 5757.140918] Key type .llcrypt unregistered [ 5757.143328] Key type ._llcrypt unregistered [ 5780.146962] Key type ._llcrypt registered [ 5780.148242] Key type .llcrypt registered [ 5780.344916] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5780.772697] Key type .llcrypt unregistered [ 5780.779224] Key type ._llcrypt unregistered [ 5797.217816] Key type ._llcrypt registered [ 5797.219891] Key type .llcrypt registered [ 5797.340987] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5798.800995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5798.859096] alg: No test for adler32 (adler32-zlib) [ 5799.939311] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 5800.225791] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5801.983213] Key type lgssc registered [ 5802.880851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5809.445868] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5809.460254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5824.993213] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5830.111792] LustreError: 185933:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [ 5856.799279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5863.583489] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5871.164827] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5877.965862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.502418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5878.610197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5883.856502] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5883.875062] LustreError: 185959:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.893404] LustreError: 185959:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5888.998581] LustreError: 185958:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5891.041096] LustreError: 185959:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5891.061768] LustreError: 185959:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5892.097785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5892.576306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5897.329167] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5905.392343] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5911.604626] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5914.482487] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5924.843638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5924.861841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.883691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5928.417734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5928.420280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5928.421675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5928.446836] Lustre: Skipped 1 previous similar message [ 5929.686586] Lustre: server umount lustre-OST0000 complete [ 5938.656204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5938.657122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5938.666574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5938.680361] Lustre: Skipped 1 previous similar message [ 5938.688876] Lustre: Skipped 1 previous similar message [ 5940.061579] Lustre: server umount lustre-MDT0000 complete [ 5943.487482] LustreError: 185940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858827 with bad export cookie 5642459694578950128 [ 5943.495221] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5943.499208] LustreError: 185940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5943.825323] Lustre: server umount lustre-MDT0001 complete [ 5956.996121] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5960.407282] Key type lgssc unregistered [ 5960.607629] LNet: 189853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5960.622860] LNetError: 189853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5960.640651] LNet: Removed LNI 192.168.201.117@tcp [ 5961.685600] Key type .llcrypt unregistered [ 5961.687923] Key type ._llcrypt unregistered [ 5988.142373] Key type ._llcrypt registered [ 5988.148296] Key type .llcrypt registered [ 5988.307549] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5989.927472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5989.963393] alg: No test for adler32 (adler32-zlib) [ 5991.191918] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 5991.472332] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5993.191533] Key type lgssc registered [ 5994.580550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6001.593388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6001.603535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6002.865417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6007.017806] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6013.412201] LustreError: 191265:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.447067] LustreError: 191265:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6016.880048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.324142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6023.187400] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6031.546986] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6038.460870] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6045.283726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6045.648728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6049.810375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6052.305979] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6060.275097] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6066.287912] Lustre: Failing over lustre-MDT0000 [ 6066.564882] Lustre: server umount lustre-MDT0000 complete [ 6067.176449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6067.196676] LustreError: 191265:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6067.212910] LustreError: 191265:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6068.416905] LustreError: 191261:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6072.290217] LustreError: 191259:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6072.308266] LustreError: 191259:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6077.409159] LustreError: 191992:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6077.430392] LustreError: 191992:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6083.551202] Lustre: 190576:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858951/real 1787858951] req@00000000984f79a5 x1874705907738624/t0(0) o400->MGC192.168.201.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787858967 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6083.570365] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6085.306402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.675148] LustreError: 191264:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6088.688227] LustreError: 191264:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 11 previous similar messages [ 6092.791512] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b689c6b8ed8333b [ 6092.803204] Lustre: MGC192.168.201.117@tcp: Connection restored to 0@lo (at 0@lo) [ 6093.104092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6093.168642] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6095.018078] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6098.016560] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6098.425793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6098.456101] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6098.519066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6103.188464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6111.294718] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6118.368716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6118.377231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6118.390508] Lustre: Skipped 2 previous similar messages [ 6118.408338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6120.406459] Lustre: server umount lustre-OST0000 complete [ 6129.121212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6129.128023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6129.147202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6129.156408] Lustre: Skipped 2 previous similar messages [ 6129.169132] Lustre: Skipped 2 previous similar messages [ 6130.490795] Lustre: server umount lustre-MDT0000 complete [ 6134.124742] LustreError: 191245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859018 with bad export cookie 10045450954621858619 [ 6134.133931] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6134.136364] LustreError: 191245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6134.506418] Lustre: server umount lustre-MDT0001 complete [ 6144.704418] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 6147.507103] Key type lgssc unregistered [ 6147.757521] LNet: 196157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6147.775954] LNetError: 196157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6148.838822] LNet: Removed LNI 192.168.201.117@tcp [ 6149.701311] Key type .llcrypt unregistered [ 6149.706419] Key type ._llcrypt unregistered [ 6173.248094] Key type ._llcrypt registered [ 6173.251452] Key type .llcrypt registered [ 6173.361813] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 6174.182930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6174.203980] alg: No test for adler32 (adler32-zlib) [ 6175.331612] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 6175.562614] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 6177.215398] Key type lgssc registered [ 6178.292551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6185.297451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6185.325420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.974947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6192.108187] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6197.232689] LustreError: 197571:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6197.261407] LustreError: 197571:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6201.918338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6202.278479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6207.872834] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6215.062948] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6220.819847] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6228.425770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6228.692938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6231.806651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6237.103806] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6244.218789] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6247.249813] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6256.216432] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6271.121703] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6283.233088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6283.249802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6283.277812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6283.285789] Lustre: Skipped 1 previous similar message [ 6288.360852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6288.376975] Lustre: Skipped 1 previous similar message [ 6289.189949] Lustre: server umount lustre-OST0000 complete [ 6293.474817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6293.490452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6293.507617] Lustre: Skipped 2 previous similar messages [ 6293.531737] Lustre: Skipped 1 previous similar message [ 6298.592800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6298.596678] Lustre: Skipped 1 previous similar message [ 6299.574866] Lustre: server umount lustre-MDT0000 complete [ 6303.276359] LustreError: 200107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859187 with bad export cookie 946734129557047419 [ 6303.295712] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6303.645982] Lustre: server umount lustre-MDT0001 complete [ 6320.731354] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 6323.952680] Key type lgssc unregistered [ 6324.215273] LNet: 201433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6324.223544] LNetError: 201433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6324.239613] LNet: Removed LNI 192.168.201.117@tcp [ 6325.097093] Key type .llcrypt unregistered [ 6325.099527] Key type ._llcrypt unregistered [ 6345.902892] Key type ._llcrypt registered [ 6345.906457] Key type .llcrypt registered [ 6346.035282] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 6347.109624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6347.206235] alg: No test for adler32 (adler32-zlib) [ 6348.239395] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 6348.446489] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 6350.095454] Key type lgssc registered [ 6351.242407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6358.575864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6358.594712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.076431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6364.143191] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6370.278810] LustreError: 202700:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6370.294117] LustreError: 202700:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6372.191516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.553652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6376.993752] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6384.684571] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6390.175897] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6395.901421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6396.153974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6401.100247] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6405.373110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6408.215886] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6414.534684] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6429.525411] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6435.112856] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6440.847937] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6448.608537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6448.619607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6448.652060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6451.680376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6451.689580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6451.699743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6454.807434] Lustre: server umount lustre-OST0000 complete [ 6459.374686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6459.394082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6459.427835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6459.440643] Lustre: Skipped 1 previous similar message [ 6461.922099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6461.932348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.044173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6467.048619] Lustre: Skipped 2 previous similar messages [ 6473.187216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6473.374745] Lustre: server umount lustre-MDT0000 complete [ 6477.281133] LustreError: 202701:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6477.288441] LustreError: 202701:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6477.650168] LustreError: 202680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859361 with bad export cookie 14285060190291274332 [ 6477.654335] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6477.660324] LustreError: 202680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6478.261819] Lustre: server umount lustre-MDT0001 complete [ 6492.250818] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 6495.581393] Key type lgssc unregistered [ 6495.884034] LNet: 207529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6495.896849] LNetError: 207529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6495.922675] LNet: Removed LNI 192.168.201.117@tcp [ 6497.099568] Key type .llcrypt unregistered [ 6497.105492] Key type ._llcrypt unregistered [ 6523.120309] Key type ._llcrypt registered [ 6523.121602] Key type .llcrypt registered [ 6523.270246] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 6524.558324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6524.611749] alg: No test for adler32 (adler32-zlib) [ 6525.773089] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 6526.166846] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 6527.847367] Key type lgssc registered [ 6529.429075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6537.753548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6537.782306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6539.457867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6545.066856] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6548.963174] LustreError: 208944:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6548.988669] LustreError: 208944:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6553.958644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.102736] LustreError: 208945:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6554.580833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6560.220113] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6569.365912] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6576.288527] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6583.480848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6583.773685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6584.850265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6589.949865] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6597.517719] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6614.503666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6614.505989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6614.524570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6614.544661] Lustre: Skipped 1 previous similar message [ 6614.570185] Lustre: Skipped 1 previous similar message [ 6617.263130] Lustre: server umount lustre-OST0000 complete [ 6621.152900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6621.155597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6621.164277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6624.741207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6624.741940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6627.083169] Lustre: server umount lustre-MDT0000 complete [ 6629.867566] LustreError: 208939:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.888653] LustreError: 208939:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6630.645496] LustreError: 210789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859514 with bad export cookie 14289685208425276744 [ 6630.656975] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6630.659444] LustreError: 210789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6631.075913] Lustre: server umount lustre-MDT0001 complete [ 6641.007432] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 6643.938280] Key type lgssc unregistered [ 6644.233782] LNet: 212631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6644.250539] LNetError: 212631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6644.274038] LNet: Removed LNI 192.168.201.117@tcp [ 6645.382862] Key type .llcrypt unregistered [ 6645.385409] Key type ._llcrypt unregistered [ 6665.948632] Key type ._llcrypt registered [ 6665.951367] Key type .llcrypt registered [ 6666.107487] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 6667.761346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6667.815771] alg: No test for adler32 (adler32-zlib) [ 6668.991723] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 6669.344977] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 6671.074873] Key type lgssc registered [ 6672.651971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6681.233276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6681.257267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6682.772975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6687.601511] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6692.396418] LustreError: 213902:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6692.426886] LustreError: 213902:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6697.177674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6697.448167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6697.626258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6702.026784] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6710.340570] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6716.461099] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6722.901782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6723.486325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6727.611208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6730.392315] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6737.525595] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6752.226250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.240485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6752.257439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6757.347976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6757.350317] Lustre: Skipped 2 previous similar messages [ 6758.143937] Lustre: server umount lustre-OST0000 complete [ 6762.465067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6762.488022] Lustre: Skipped 1 previous similar message [ 6762.493749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6762.503417] Lustre: Skipped 1 previous similar message [ 6767.586689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6767.598992] Lustre: Skipped 1 previous similar message [ 6768.844995] Lustre: server umount lustre-MDT0000 complete [ 6772.718928] LustreError: 213897:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6772.726815] LustreError: 213880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859656 with bad export cookie 17291695980466139698 [ 6772.747947] LustreError: 213897:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6772.763207] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6772.765758] LustreError: 213880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6773.445051] Lustre: server umount lustre-MDT0001 complete [ 6784.292751] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 6786.886437] Key type lgssc unregistered [ 6787.128403] LNet: 217585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6787.137528] LNetError: 217585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6788.199688] LNet: Removed LNI 192.168.201.117@tcp [ 6789.127714] Key type .llcrypt unregistered [ 6789.129860] Key type ._llcrypt unregistered [ 6806.622469] Key type ._llcrypt registered [ 6806.624870] Key type .llcrypt registered [ 6806.800619] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 6807.719812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6807.738927] alg: No test for adler32 (adler32-zlib) [ 6808.795207] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 6809.010504] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 6810.656381] Key type lgssc registered [ 6811.624515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6820.216868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6820.246533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6822.065173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6827.836896] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6832.615782] LustreError: 218844:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6832.635024] LustreError: 218844:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6833.651034] LustreError: 218845:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.760193] LustreError: 218844:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6839.974639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.277234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6845.141632] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6851.653809] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6857.663568] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6865.426793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6865.687604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6868.816560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6870.947881] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6878.649450] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6887.357899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6887.406883] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6887.653217] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6894.203308] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6901.752935] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6903.061870] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6914.813463] Lustre: Permanently deactivating lustre-OST0001 [ 6914.827193] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6921.839523] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6928.101301] Lustre: Permanently reactivating lustre-OST0001 [ 6928.116922] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6928.121376] Lustre: Skipped 2 previous similar messages [ 6929.189702] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6929.204714] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6929.228661] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6929.266444] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6929.335820] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6939.618877] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6939.625413] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6939.631391] LustreError: Skipped 1 previous similar message [ 6939.632509] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6939.695078] Lustre: Skipped 2 previous similar messages [ 6940.820489] Lustre: server umount lustre-OST0001 complete [ 6941.682227] LustreError: 220668:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6941.694966] LustreError: 220668:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6945.778307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6945.790144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6945.807332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6945.815155] Lustre: Skipped 1 previous similar message [ 6946.791118] LustreError: 220669:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.813449] LustreError: 220669:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6950.907615] Lustre: server umount lustre-OST0000 complete [ 6957.030861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6957.053846] Lustre: Skipped 2 previous similar messages [ 6957.060602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6957.066561] Lustre: Skipped 3 previous similar messages [ 6961.603746] Lustre: server umount lustre-MDT0000 complete [ 6962.147306] LustreError: 218844:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6962.167235] LustreError: 218844:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6965.416467] LustreError: 221683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859849 with bad export cookie 5803885105204259509 [ 6965.419328] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6965.425914] LustreError: 221683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6965.788246] Lustre: server umount lustre-MDT0001 complete [ 6977.686928] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 6980.436273] Key type lgssc unregistered [ 6980.784719] LNet: 223893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6980.796632] LNetError: 223893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6981.861122] LNet: Removed LNI 192.168.201.117@tcp [ 6983.085473] Key type .llcrypt unregistered [ 6983.088269] Key type ._llcrypt unregistered [ 7028.452597] Key type ._llcrypt registered [ 7028.455771] Key type .llcrypt registered [ 7028.587823] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7030.049181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7030.190573] alg: No test for adler32 (adler32-zlib) [ 7031.449462] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 7031.892925] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 7033.615202] Key type lgssc registered [ 7034.895447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7042.713859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7042.755239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.163441] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7044.205307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7044.424443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7050.363648] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7060.356968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.429203] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7060.459776] Lustre: 226099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7060.819678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7066.400420] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7074.732130] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7081.208407] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7088.920776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7089.220294] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7089.512093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7096.247515] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7096.560338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7105.260509] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7108.247851] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7116.279687] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7126.792819] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7135.702726] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7152.232767] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7152.241472] Lustre: Skipped 1 previous similar message [ 7170.851267] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7170.861268] Lustre: Skipped 1 previous similar message [ 7204.748919] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7204.754312] Lustre: Skipped 4 previous similar messages [ 7281.871881] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7281.883973] Lustre: Skipped 8 previous similar messages [ 7290.852662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7290.867234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7290.893287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7291.373851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7291.374956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7291.401593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7291.417629] Lustre: Skipped 1 previous similar message [ 7294.774328] Lustre: server umount lustre-OST0000 complete [ 7301.610185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.614909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7301.635390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7301.672275] Lustre: Skipped 1 previous similar message [ 7305.149643] Lustre: server umount lustre-MDT0000 complete [ 7309.259450] LustreError: 226092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860193 with bad export cookie 4895987088703914308 [ 7309.266602] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7309.267757] LustreError: 226092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7309.938257] Lustre: server umount lustre-MDT0001 complete [ 7322.695397] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 7325.804165] Key type lgssc unregistered [ 7326.075870] LNet: 230923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7326.092296] LNetError: 230923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7326.106279] LNet: Removed LNI 192.168.201.117@tcp [ 7326.984278] Key type .llcrypt unregistered [ 7326.985807] Key type ._llcrypt unregistered [ 7351.169776] Key type ._llcrypt registered [ 7351.171603] Key type .llcrypt registered [ 7351.262598] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7352.141331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7352.212445] alg: No test for adler32 (adler32-zlib) [ 7353.149494] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 7353.370174] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 7355.007268] Key type lgssc registered [ 7356.217368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7364.419027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7364.434824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.055680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7371.923549] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7376.389206] LustreError: 232336:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7376.428650] LustreError: 232336:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7381.474770] LustreError: 232337:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7382.516321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.248932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7388.714433] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7396.370604] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7402.896450] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7410.160467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7410.584123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7411.643785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7416.661467] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7424.966692] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7444.449802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7444.454140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7444.458199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7444.471936] Lustre: Skipped 1 previous similar message [ 7449.575245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7449.593525] Lustre: Skipped 1 previous similar message [ 7450.609607] Lustre: server umount lustre-OST0000 complete [ 7454.688811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7454.691988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7454.732190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7454.734845] Lustre: Skipped 1 previous similar message [ 7459.811947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7459.819145] Lustre: Skipped 1 previous similar message [ 7460.615629] Lustre: server umount lustre-MDT0000 complete [ 7464.928299] LustreError: 233065:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.943415] LustreError: 233065:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7465.111659] LustreError: 234916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860349 with bad export cookie 3894511333631121967 [ 7465.124161] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7465.126749] LustreError: 234916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7465.608546] Lustre: server umount lustre-MDT0001 complete [ 7479.457066] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 7482.288613] Key type lgssc unregistered [ 7482.647077] LNet: 236029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7482.655138] LNetError: 236029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7482.678914] LNet: Removed LNI 192.168.201.117@tcp [ 7483.316397] Key type .llcrypt unregistered [ 7483.320961] Key type ._llcrypt unregistered [ 7507.625417] Key type ._llcrypt registered [ 7507.628180] Key type .llcrypt registered [ 7507.858969] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7508.962948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7509.086283] alg: No test for adler32 (adler32-zlib) [ 7510.642719] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 7511.119208] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 7512.935271] Key type lgssc registered [ 7514.046850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7785.654714] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 7788.541048] Key type lgssc unregistered [ 7788.915469] LNet: 238063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7788.924719] LNetError: 238063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7788.940913] LNet: Removed LNI 192.168.201.117@tcp [ 7789.935589] Key type .llcrypt unregistered [ 7789.937221] Key type ._llcrypt unregistered [ 7816.993903] Key type ._llcrypt registered [ 7816.997442] Key type .llcrypt registered [ 7817.078188] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 7832.105317] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7833.155342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7833.192162] alg: No test for adler32 (adler32-zlib) [ 7834.277470] Lustre: Lustre: Build Version: 2.17.57_84_g466c428 [ 7834.526442] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 7836.175165] Key type lgssc registered [ 7838.023419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7870.423831] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7619 sec ========== 15:59:13 (1787860753) [ 7872.014760] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:59:15 (1787860755) === [ 7875.881511] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:59:19 (1787860759) === [ 7921.234616] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 7924.489397] Key type lgssc unregistered [ 7924.857679] LNet: 243303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7924.861356] LNetError: 243303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7924.880367] LNet: Removed LNI 192.168.201.117@tcp [ 7925.916759] Key type .llcrypt unregistered [ 7925.919802] Key type ._llcrypt unregistered