[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 467982903 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002399] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009154] LSM: Security Framework initializing [ 0.010099] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.012095] *** VALIDATE selinux *** [ 0.022427] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026814] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029077] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030097] *** VALIDATE tmpfs *** [ 0.032039] *** VALIDATE proc *** [ 0.033217] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035238] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042592] debug: unmapping init [mem 0xffffffffb3259000-0xffffffffb3260fff] [ 0.044917] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045877] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051008] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053299] rcu: Hierarchical SRCU implementation. [ 0.055377] smp: Bringing up secondary CPUs ... [ 0.056585] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.060410] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137851] node 0 deferred pages initialised in 72ms [ 0.139102] devtmpfs: initialized [ 0.140221] x86/mm: Memory block size: 128MB [ 0.142396] gcov: version magic: 0x41383552 [ 0.144206] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146252] pinctrl core: initialized pinctrl subsystem [ 0.147198] [ 0.148009] ************************************************************* [ 0.149009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150010] ** ** [ 0.151008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.152009] ** ** [ 0.153013] ** This means that this kernel is built to expose internal ** [ 0.154023] ** IOMMU data structures, which may compromise security on ** [ 0.155013] ** your system. ** [ 0.156011] ** ** [ 0.157012] ** If you see this message and you are not debugging the ** [ 0.158012] ** kernel, report this immediately to your vendor! ** [ 0.159011] ** ** [ 0.160013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161010] ************************************************************* [ 0.162696] NET: Registered protocol family 16 [ 0.163445] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.164066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.165052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.166369] cpuidle: using governor menu [ 0.168651] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170413] PCI: Using configuration type 1 for base access [ 0.173130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.181180] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188081] cryptd: max_cpu_qlen set to 1000 [ 0.190214] ACPI: Added _OSI(Module Device) [ 0.191015] ACPI: Added _OSI(Processor Device) [ 0.192010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.193010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.196809] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.199464] ACPI: Interpreter enabled [ 0.201057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.202012] ACPI: Using IOAPIC for interrupt routing [ 0.204113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.206399] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.216886] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.219036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.224085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.229923] acpiphp: Slot [2] registered [ 0.231182] acpiphp: Slot [5] registered [ 0.232102] acpiphp: Slot [6] registered [ 0.234117] acpiphp: Slot [7] registered [ 0.235103] acpiphp: Slot [8] registered [ 0.237090] acpiphp: Slot [9] registered [ 0.238101] acpiphp: Slot [10] registered [ 0.239104] acpiphp: Slot [3] registered [ 0.241087] acpiphp: Slot [4] registered [ 0.242104] acpiphp: Slot [11] registered [ 0.244156] acpiphp: Slot [12] registered [ 0.245088] acpiphp: Slot [13] registered [ 0.246100] acpiphp: Slot [14] registered [ 0.248076] acpiphp: Slot [15] registered [ 0.249077] acpiphp: Slot [16] registered [ 0.250094] acpiphp: Slot [17] registered [ 0.252089] acpiphp: Slot [18] registered [ 0.253079] acpiphp: Slot [19] registered [ 0.254101] acpiphp: Slot [20] registered [ 0.256087] acpiphp: Slot [21] registered [ 0.257086] acpiphp: Slot [22] registered [ 0.258153] acpiphp: Slot [23] registered [ 0.260070] acpiphp: Slot [24] registered [ 0.261126] acpiphp: Slot [25] registered [ 0.263090] acpiphp: Slot [26] registered [ 0.265107] acpiphp: Slot [27] registered [ 0.266099] acpiphp: Slot [28] registered [ 0.268101] acpiphp: Slot [29] registered [ 0.269147] acpiphp: Slot [30] registered [ 0.271072] acpiphp: Slot [31] registered [ 0.272039] PCI host bridge to bus 0000:00 [ 0.273012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.276023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.279023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.282052] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.285022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.288022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.289146] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.293109] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.296082] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.306015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.312061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.315017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.317015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.319015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.321586] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.324074] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.327047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.331006] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.337013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.349015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.354798] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.360722] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.368018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.374013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.387014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.396000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.402013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.410015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.426019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.437290] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.443016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.450014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.466009] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.478469] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.486015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.494016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.516017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.524406] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.531015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.539016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.560016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.574027] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.583013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.588017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.605013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.614095] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.616255] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.618379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.621354] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.623192] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.628258] iommu: Default domain type: Passthrough [ 0.631429] SCSI subsystem initialized [ 0.633116] ACPI: bus type USB registered [ 0.635117] usbcore: registered new interface driver usbfs [ 0.637173] usbcore: registered new interface driver hub [ 0.639076] usbcore: registered new device driver usb [ 0.641173] pps_core: LinuxPPS API ver. 1 registered [ 0.643011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.647057] PTP clock support registered [ 0.649158] EDAC MC: Ver: 3.0.0 [ 0.651103] PCI: Using ACPI for IRQ routing [ 0.652821] NetLabel: Initializing [ 0.654010] NetLabel: domain hash size = 128 [ 0.656012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.658091] NetLabel: unlabeled traffic allowed by default [ 0.661187] vgaarb: loaded [ 0.663284] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.665014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.672719] clocksource: Switched to clocksource kvm-clock [ 0.775644] VFS: Disk quotas dquot_6.6.0 [ 0.777784] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.782108] *** VALIDATE ramfs *** [ 0.784110] *** VALIDATE hugetlbfs *** [ 0.786459] pnp: PnP ACPI init [ 0.789558] pnp: PnP ACPI: found 6 devices [ 0.805652] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.809791] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.812136] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.814290] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.817170] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.819931] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.823371] NET: Registered protocol family 2 [ 0.825674] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.830217] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.833787] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.839039] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.843298] TCP: Hash tables configured (established 65536 bind 65536) [ 0.847218] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.850609] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.853578] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.856741] NET: Registered protocol family 1 [ 0.861296] RPC: Registered named UNIX socket transport module. [ 0.863956] RPC: Registered udp transport module. [ 0.865959] RPC: Registered tcp transport module. [ 0.868175] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.872759] NET: Registered protocol family 44 [ 0.874581] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.876770] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.878911] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.881412] PCI: CLS 0 bytes, default 64 [ 0.883223] Unpacking initramfs... [ 2.275194] debug: unmapping init [mem 0xffff9d4e3cc54000-0xffff9d4e3ffbffff] [ 2.279482] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.283232] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.286235] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.782937] Initialise system trusted keyrings [ 2.787210] Key type blacklist registered [ 2.790324] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.799616] zbud: loaded [ 2.804332] *** VALIDATE nfs *** [ 2.806374] *** VALIDATE nfs4 *** [ 2.808637] pstore: using deflate compression [ 2.814306] Platform Keyring initialized [ 2.947577] NET: Registered protocol family 38 [ 2.950344] Key type asymmetric registered [ 2.952887] Asymmetric key parser 'x509' registered [ 2.955594] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.960163] io scheduler mq-deadline registered [ 2.962613] io scheduler kyber registered [ 2.964761] io scheduler bfq registered [ 2.967157] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.971246] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.975088] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.979102] ACPI: Power Button [PWRF] [ 2.985782] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.992747] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.011390] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.021339] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.040411] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.068832] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.098716] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.104210] Non-volatile memory driver v1.3 [ 3.106327] Linux agpgart interface v0.103 [ 3.138230] virtio_blk virtio1: [vda] 149568 512-byte logical blocks (76.6 MB/73.0 MiB) [ 3.141413] vda: detected capacity change from 0 to 76578816 [ 3.154392] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.157042] vdb: detected capacity change from 0 to 1073741824 [ 3.172263] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.175191] vdc: detected capacity change from 0 to 2621440000 [ 3.192725] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.197959] vdd: detected capacity change from 0 to 2621440000 [ 3.213782] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.216787] vde: detected capacity change from 0 to 4294967296 [ 3.230235] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.235450] vdf: detected capacity change from 0 to 4294967296 [ 3.246341] libphy: Fixed MDIO Bus: probed [ 3.250863] usbcore: registered new interface driver usbserial_generic [ 3.253619] usbserial: USB Serial support registered for generic [ 3.256355] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.261871] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.265064] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.268157] mousedev: PS/2 mouse device common for all mice [ 3.272259] rtc_cmos 00:05: RTC can wake from S4 [ 3.275061] rtc_cmos 00:05: registered as rtc0 [ 3.276145] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.276635] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.282197] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.282690] intel_pstate: CPU model not supported [ 3.287338] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.290622] hid: raw HID events driver (C) Jiri Kosina [ 3.292166] usbcore: registered new interface driver usbhid [ 3.293673] usbhid: USB HID core driver [ 3.294948] drop_monitor: Initializing network drop monitor service [ 3.296726] Initializing XFRM netlink socket [ 3.298340] NET: Registered protocol family 10 [ 3.300629] Segment Routing with IPv6 [ 3.301670] NET: Registered protocol family 17 [ 3.303347] mpls_gso: MPLS GSO support [ 3.309647] RAS: Correctable Errors collector initialized. [ 3.311457] AVX version of gcm_enc/dec engaged. [ 3.312578] AES CTR mode by8 optimization enabled [ 3.381714] sched_clock: Marking stable (3381690368, 0)->(4339712930, -958022562) [ 3.384524] registered taskstats version 1 [ 3.386064] Loading compiled-in X.509 certificates [ 3.387466] zswap: loaded using pool lzo/zbud [ 3.408819] Key type big_key registered [ 3.419967] Key type encrypted registered [ 3.421897] ima: No TPM chip found, activating TPM-bypass! [ 3.423781] ima: Allocated hash algorithm: sha1 [ 3.425217] ima: No architecture policies found [ 3.426621] evm: Initialising EVM extended attributes: [ 3.428687] evm: security.selinux [ 3.429932] evm: security.ima [ 3.430778] evm: security.capability [ 3.432182] evm: HMAC attrs: 0x1 [ 3.434474] rtc_cmos 00:05: setting system clock to 2026-08-22 05:45:47 UTC (1787377547) [ 3.441240] debug: unmapping init [mem 0xffffffffb4203000-0xffffffffb43fffff] [ 3.444346] debug: unmapping init [mem 0xffffffffb2f82000-0xffffffffb3258fff] [ 3.449367] Write protecting the kernel read-only data: 28672k [ 3.452328] debug: unmapping init [mem 0xffffffffb1603000-0xffffffffb17fffff] [ 3.455028] debug: unmapping init [mem 0xffffffffb1f14000-0xffffffffb1ffffff] [ 3.488555] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.499265] systemd[1]: Detected virtualization kvm. [ 3.501187] systemd[1]: Detected architecture x86-64. [ 3.503246] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.534161] systemd[1]: No hostname configured. [ 3.536041] systemd[1]: Set hostname to . [ 3.538319] random: systemd: uninitialized urandom read (16 bytes read) [ 3.541064] systemd[1]: Initializing machine ID from random generator. [ 3.591035] random: ln: uninitialized urandom read (6 bytes read) [ 3.661192] random: systemd: uninitialized urandom read (16 bytes read) [ 3.663996] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.668869] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.673620] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.302366] device-mapper: uevent: version 1.0.3 [ 4.304919] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.040717] random: fast init done [ 5.061621] virtio_net virtio0 ens2: renamed from eth0 [ 5.079872] scsi host0: ata_piix [ 5.098299] scsi host1: ata_piix [ 5.112139] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.115171] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.380629] dracut-initqueue[592]: RTNETLINK answers: File exists [ 9.937271] random: crng init done [ 9.938806] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.287668] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.356677] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.620485] SELinux: Disabled at runtime. [ 11.681107] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.694786] systemd[1]: Detected virtualization kvm. [ 11.696989] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.166430] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.169596] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.175216] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.178459] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.181432] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.190067] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.198668] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED[ 12.375214] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.574420] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.889197] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.918579] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.989516] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.002328] EDAC sbridge: Ver: 1.1.2 [ 14.712645] Key type dns_resolver registered [ 15.018102] NFS: Registering the id_resolver key type [ 15.020594] Key type id_resolver registered [ 15.022714] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg316-server login: [ 41.090963] libcfs: loading out-of-tree module taints kernel. [ 41.111758] Key type ._llcrypt registered [ 41.113435] Key type .llcrypt registered [ 41.165566] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 62.417480] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 64.228236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.242535] alg: No test for adler32 (adler32-zlib) [ 65.893717] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 66.959826] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 68.852209] Key type lgssc registered [ 71.218150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.430569] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 139.787076] hrtimer: interrupt took 7064971 ns [ 141.213158] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 155.843632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 155.911819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.077200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 157.120301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 157.210677] Lustre: lustre-MDT0000: new disk, initializing [ 157.303167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.341587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.443226] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.938153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 175.122471] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 175.179923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 175.191204] Lustre: Skipped 1 previous similar message [ 175.346700] Lustre: lustre-MDT0001: new disk, initializing [ 175.425264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 175.473359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 175.487256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.188401] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.511708] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 196.172529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.454216] Lustre: lustre-OST0000: new disk, initializing [ 196.457422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.462943] Lustre: 8446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.518942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.429811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 197.440941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 197.512955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 202.689857] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.278938] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.420778] Lustre: lustre-OST0001: new disk, initializing [ 216.423851] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.428926] Lustre: 9517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.495105] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.112691] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.835803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.849364] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.935063] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 235.732166] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.466264] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 250.675276] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing check_logdir /tmp/testlogs/ [ 255.333103] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing yml_node [ 260.256963] Lustre: DEBUG MARKER: Client: 2.17.54.117 [ 262.598921] Lustre: DEBUG MARKER: MDS: 2.17.54.117 [ 265.266526] Lustre: DEBUG MARKER: OSS: 2.17.54.117 [ 267.057538] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 01:50:10 EDT 2026 [ 283.895202] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 285.167629] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.025034] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.028530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.048616] Lustre: Skipped 3 previous similar messages [ 301.052216] Lustre: Skipped 2 previous similar messages [ 306.146673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.161986] Lustre: Skipped 4 previous similar messages [ 311.266347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.278868] Lustre: Skipped 3 previous similar messages [ 315.359470] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 317.517997] Lustre: server umount lustre-MDT0000 complete [ 320.791700] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377864 with bad export cookie 8712263526309128025 [ 320.798404] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.804250] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 321.503901] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.504115] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.504993] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.514889] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 321.521942] Lustre: Skipped 1 previous similar message [ 325.562198] LustreError: 10344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.566161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.578301] LustreError: 10344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 325.609450] Lustre: Skipped 2 previous similar messages [ 330.720349] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.752133] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 335.327900] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 335.848720] LustreError: 10344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.875378] LustreError: 10344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 336.069431] Lustre: server umount lustre-MDT0001 complete [ 341.170036] Lustre: server umount lustre-OST0000 complete [ 346.475439] Lustre: server umount lustre-OST0001 complete [ 353.796330] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 363.568384] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 402.648451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.015086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 403.059086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 403.177791] Lustre: lustre-MDT0000: new disk, initializing [ 403.239681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.260347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 408.045581] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 420.834146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.074234] Lustre: 15076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 421.169954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 421.182313] Lustre: Skipped 1 previous similar message [ 421.340103] Lustre: lustre-MDT0001: new disk, initializing [ 421.519581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.592397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.609267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 427.028025] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 437.820656] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 444.536631] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 451.500433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 451.877408] Lustre: lustre-OST0000: new disk, initializing [ 451.881637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 451.891678] Lustre: 17044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 452.004337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.639865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 453.646494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 453.769734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 458.548307] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 467.205092] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 471.978856] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 472.184888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 476.662793] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 476.842932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 479.200930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 479.204086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.216909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 479.227317] Lustre: Skipped 1 previous similar message [ 479.240522] Lustre: Skipped 2 previous similar messages [ 484.618600] Lustre: server umount lustre-OST0000 complete [ 489.441892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.461400] Lustre: Skipped 1 previous similar message [ 494.735756] Lustre: server umount lustre-MDT0000 complete [ 498.316041] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378042 with bad export cookie 8712263526309129551 [ 498.319851] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 498.333096] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 498.792325] Lustre: server umount lustre-MDT0001 complete [ 502.104411] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:54:05 (1787378045) [ 509.008295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.372979] LustreError: 19309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.459146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.946988] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 519.648596] LustreError: 19310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.670997] LustreError: 19310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 521.921753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.502847] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 534.510933] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 541.171932] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 550.157564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 557.184988] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 564.907412] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 577.002603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 577.019905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.039566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 577.042247] Lustre: Skipped 7 previous similar messages [ 582.392545] Lustre: server umount lustre-OST0000 complete [ 587.746211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.755564] Lustre: Skipped 2 previous similar messages [ 592.446275] Lustre: server umount lustre-MDT0000 complete [ 592.866500] LustreError: 19310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.890380] LustreError: 19310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 595.918438] LustreError: 19292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378139 with bad export cookie 8712263526309130433 [ 595.939744] LustreError: 19292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 595.943696] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.248905] Lustre: server umount lustre-MDT0001 complete [ 608.724285] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 613.359579] Key type lgssc unregistered [ 613.808229] LNet: 22844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.827696] LNetError: 22844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.897105] LNet: Removed LNI 192.168.203.116@tcp [ 616.103151] Key type .llcrypt unregistered [ 616.104620] Key type ._llcrypt unregistered [ 643.203364] Key type ._llcrypt registered [ 643.205413] Key type .llcrypt registered [ 643.284867] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 644.104554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 644.115782] alg: No test for adler32 (adler32-zlib) [ 645.189791] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 645.564406] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 647.239413] Key type lgssc registered [ 648.319902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 656.592303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 656.630320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.199828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.595285] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 668.643888] LustreError: 24262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.670320] LustreError: 24262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 671.647564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.008196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 676.680351] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 683.913686] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 690.046895] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 697.462839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 697.772769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 703.872767] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 704.018531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 710.793189] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 728.556461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 728.570672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 728.588668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 729.023255] Lustre: server umount lustre-OST0000 complete [ 733.665370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 733.682677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.704233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 734.703501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.720736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 734.723828] Lustre: Skipped 1 previous similar message [ 739.808422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 744.026863] Lustre: server umount lustre-MDT0000 complete [ 744.936894] LustreError: 24989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 744.974230] LustreError: 24989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 748.336820] LustreError: 24243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378292 with bad export cookie 6622240696491634164 [ 748.360113] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 749.091750] Lustre: server umount lustre-MDT0001 complete [ 762.051327] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 764.772437] Key type lgssc unregistered [ 765.187184] LNet: 28237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.191902] LNetError: 28237:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.226252] LNet: Removed LNI 192.168.203.116@tcp [ 766.241333] Key type .llcrypt unregistered [ 766.244318] Key type ._llcrypt unregistered [ 792.166411] Key type ._llcrypt registered [ 792.171812] Key type .llcrypt registered [ 792.328626] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 793.711890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 793.745545] alg: No test for adler32 (adler32-zlib) [ 794.913953] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 795.153165] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 796.815137] Key type lgssc registered [ 797.921260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 806.699872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 806.741606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.258354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 813.135885] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 818.658544] LustreError: 29652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 818.686148] LustreError: 29652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 822.518154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.945391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 827.671478] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 835.481744] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 841.380568] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 855.231198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 855.829911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.964248] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 873.766609] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 878.391371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 887.776608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 887.790902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.819731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 888.801961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 888.803296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.806060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 893.924126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 893.941382] Lustre: Skipped 2 previous similar messages [ 899.039516] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 899.047160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 899.081851] Lustre: Skipped 1 previous similar message [ 899.300382] Lustre: server umount lustre-OST0000 complete [ 904.161225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.193508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 904.206861] Lustre: Skipped 1 previous similar message [ 909.896589] Lustre: server umount lustre-MDT0000 complete [ 914.402000] LustreError: 29652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.428537] LustreError: 29652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 914.775442] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378458 with bad export cookie 13077832547675464168 [ 914.785485] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 914.786124] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 915.189708] Lustre: server umount lustre-MDT0001 complete [ 929.917959] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 933.121961] Key type lgssc unregistered [ 933.471757] LNet: 33546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.487616] LNetError: 33546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.514230] LNet: Removed LNI 192.168.203.116@tcp [ 934.667780] Key type .llcrypt unregistered [ 934.672732] Key type ._llcrypt unregistered [ 959.729271] Key type ._llcrypt registered [ 959.731216] Key type .llcrypt registered [ 959.828738] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 961.205773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 961.220625] alg: No test for adler32 (adler32-zlib) [ 962.312907] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 962.540457] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 964.207191] Key type lgssc registered [ 965.465482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 974.547124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 974.570267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.143834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 980.272984] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 986.599989] LustreError: 34958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 986.630158] LustreError: 34958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 988.982945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.419748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 993.976136] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1002.260761] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1008.127580] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1014.861448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1015.326436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.883639] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1022.513486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1029.558652] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1042.912646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1042.924502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.929982] LustreError: Skipped 1 previous similar message [ 1042.954078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1044.959910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1044.960169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1044.971751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.990788] Lustre: Skipped 1 previous similar message [ 1048.712296] Lustre: server umount lustre-OST0000 complete [ 1055.203931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.210048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.227638] Lustre: Skipped 1 previous similar message [ 1055.251472] Lustre: Skipped 1 previous similar message [ 1059.025327] Lustre: server umount lustre-MDT0000 complete [ 1060.326543] LustreError: 34959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1060.353669] LustreError: 34959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1062.751815] LustreError: 34939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378606 with bad export cookie 1776500912425760281 [ 1062.759184] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.768353] LustreError: 34939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1063.180845] Lustre: server umount lustre-MDT0001 complete [ 1074.645763] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1077.662890] Key type lgssc unregistered [ 1077.998848] LNet: 38516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.003538] LNetError: 38516:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1079.085956] LNet: Removed LNI 192.168.203.116@tcp [ 1080.212371] Key type .llcrypt unregistered [ 1080.215139] Key type ._llcrypt unregistered [ 1103.918926] Key type ._llcrypt registered [ 1103.922885] Key type .llcrypt registered [ 1104.053938] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1104.884995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1105.014884] alg: No test for adler32 (adler32-zlib) [ 1106.131944] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 1106.410641] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1108.140181] Key type lgssc registered [ 1109.137300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.380534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1116.412868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.861237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1122.496996] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1127.396705] LustreError: 39931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.417032] LustreError: 39931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1131.555424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.065689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1136.504951] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1144.603966] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1151.246886] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1158.464631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1158.839173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.939509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1164.331866] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1172.238779] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1182.697909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.699091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 1182.725823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1182.728226] LustreError: Skipped 1 previous similar message [ 1182.746205] Lustre: Skipped 1 previous similar message [ 1184.252744] Lustre: lustre-OST0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1187.305814] Lustre: server umount lustre-OST0000 complete [ 1207.267523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.273235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.278381] Lustre: Skipped 2 previous similar messages [ 1207.292142] Lustre: Skipped 1 previous similar message [ 1212.384150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1213.457644] Lustre: server umount lustre-MDT0000 complete [ 1217.288101] LustreError: 39911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378761 with bad export cookie 13213461195233093294 [ 1217.304504] LustreError: 40662:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 1217.304567] LustreError: 39911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1217.305054] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1217.329465] LustreError: 40662:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 1223.648287] Lustre: server umount lustre-MDT0001 complete [ 1235.084759] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1237.423752] Key type lgssc unregistered [ 1237.780924] LNet: 43642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.801452] LNetError: 43642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.816425] LNet: Removed LNI 192.168.203.116@tcp [ 1238.523121] Key type .llcrypt unregistered [ 1238.524977] Key type ._llcrypt unregistered [ 1260.695338] Key type ._llcrypt registered [ 1260.704153] Key type .llcrypt registered [ 1260.819841] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1261.987491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1262.013883] alg: No test for adler32 (adler32-zlib) [ 1263.095526] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 1263.368062] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1265.096284] Key type lgssc registered [ 1266.376525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1275.244358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1275.267860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.735760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.887919] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1286.136775] LustreError: 45051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.155775] LustreError: 45051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1290.487751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.962671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1295.319777] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1302.872613] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1308.965444] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1315.996495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1316.470980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1321.602128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1322.957341] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1331.025931] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1341.920386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1341.932396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.944458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1344.512952] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1346.530864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.532199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.543355] Lustre: Skipped 1 previous similar message [ 1346.563920] Lustre: Skipped 1 previous similar message [ 1347.939274] Lustre: server umount lustre-MDT0000 complete [ 1349.590242] LustreError: 45047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.660994] LustreError: 45051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.686232] LustreError: 45051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1351.790458] LustreError: 46896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378895 with bad export cookie 1034730843255897061 [ 1351.810504] LustreError: 46896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1351.811281] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1354.709348] LustreError: 46078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.750137] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 1354.758279] Lustre: Skipped 1 previous similar message [ 1356.769111] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.119532] Lustre: server umount lustre-MDT0001 complete [ 1373.193361] Lustre: server umount lustre-OST0000 complete [ 1391.024808] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1393.912642] Key type lgssc unregistered [ 1394.178555] LNet: 48912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.187868] LNetError: 48912:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.211046] LNet: Removed LNI 192.168.203.116@tcp [ 1394.979939] Key type .llcrypt unregistered [ 1394.983302] Key type ._llcrypt unregistered [ 1419.152941] Key type ._llcrypt registered [ 1419.155369] Key type .llcrypt registered [ 1419.322530] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1420.318332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1420.356770] alg: No test for adler32 (adler32-zlib) [ 1421.424390] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 1421.772414] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1423.447292] Key type lgssc registered [ 1424.836313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1433.910316] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1433.945670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1449.568262] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1454.688445] LustreError: 50299:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 1481.375213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1488.488091] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1497.407393] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1664.991325] Lustre: server umount lustre-OST0000 complete [ 1680.215149] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1682.658512] Key type lgssc unregistered [ 1682.897080] LNet: 51976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.911916] LNetError: 51976:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.935040] LNet: Removed LNI 192.168.203.116@tcp [ 1683.721766] Key type .llcrypt unregistered [ 1683.724934] Key type ._llcrypt unregistered [ 1704.787749] Key type ._llcrypt registered [ 1704.791049] Key type .llcrypt registered [ 1704.924812] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1706.303942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1706.333760] alg: No test for adler32 (adler32-zlib) [ 1707.483680] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 1707.731314] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1709.408942] Key type lgssc registered [ 1710.462161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1717.735557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1717.747162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.169896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.316846] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1729.506629] LustreError: 53381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1729.518487] LustreError: 53381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1732.000224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.429227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1737.077356] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1745.275968] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1751.782581] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1757.584209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1757.886036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1763.179666] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1764.013358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1771.010683] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1774.562270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1774.562586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1774.576426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1777.634019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.758133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.779923] Lustre: Skipped 1 previous similar message [ 1787.360212] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1787.516116] Lustre: server umount lustre-OST0000 complete [ 1792.994876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.998352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1793.016627] Lustre: Skipped 1 previous similar message [ 1793.023570] Lustre: Skipped 1 previous similar message [ 1798.113740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1798.122088] Lustre: Skipped 2 previous similar messages [ 1805.279255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1805.440095] Lustre: server umount lustre-MDT0000 complete [ 1808.362760] LustreError: 53376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1808.390851] LustreError: 53376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1809.443374] LustreError: 53363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379353 with bad export cookie 9979691951608234276 [ 1809.469140] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1809.881689] Lustre: server umount lustre-MDT0001 complete [ 1821.628453] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1824.487988] Key type lgssc unregistered [ 1824.831876] LNet: 56916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.839161] LNetError: 56916:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.848633] LNet: Removed LNI 192.168.203.116@tcp [ 1825.708119] Key type .llcrypt unregistered [ 1825.714394] Key type ._llcrypt unregistered [ 1848.728318] Key type ._llcrypt registered [ 1848.733221] Key type .llcrypt registered [ 1848.861606] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1850.010995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.030942] alg: No test for adler32 (adler32-zlib) [ 1851.094808] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 1851.302371] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1853.002550] Key type lgssc registered [ 1854.107138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1862.039845] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1862.053028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.599421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1882.720329] LustreError: 58304:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 1909.408973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.393203] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1924.942328] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1932.058237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.564950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.622504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1937.085989] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1937.906272] LustreError: 58330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.920983] LustreError: 58330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1943.011572] LustreError: 58329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.045099] LustreError: 58330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.067159] LustreError: 58330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1946.521852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.870241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1951.504762] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1959.470341] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1966.349313] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1972.724630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.732747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.751729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.763284] Lustre: Skipped 1 previous similar message [ 1975.063353] Lustre: server umount lustre-OST0000 complete [ 2002.414523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.424926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.438824] Lustre: Skipped 2 previous similar messages [ 2002.457569] Lustre: Skipped 1 previous similar message [ 2007.520699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.538198] Lustre: Skipped 1 previous similar message [ 2012.641344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2012.656471] Lustre: Skipped 1 previous similar message [ 2013.151155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2013.310380] Lustre: server umount lustre-MDT0000 complete [ 2016.558230] LustreError: 58312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379560 with bad export cookie 4369587337003295819 [ 2016.569539] LustreError: 58312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2016.570402] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.033545] Lustre: server umount lustre-MDT0001 complete [ 2033.846738] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2036.849235] Key type lgssc unregistered [ 2037.161113] LNet: 62050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.173439] LNetError: 62050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.184460] LNet: Removed LNI 192.168.203.116@tcp [ 2037.967833] Key type .llcrypt unregistered [ 2037.969802] Key type ._llcrypt unregistered [ 2061.735914] Key type ._llcrypt registered [ 2061.740873] Key type .llcrypt registered [ 2061.868877] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2063.177427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2063.260451] alg: No test for adler32 (adler32-zlib) [ 2064.432370] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 2064.700593] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2066.455150] Key type lgssc registered [ 2067.753812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2075.400497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2075.466602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.057920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.515183] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2087.393820] LustreError: 63482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.407288] LustreError: 63482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2089.232931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.581453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2093.220766] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2100.809802] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2106.513556] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2113.125681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2113.427894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.598803] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2122.674649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2126.755571] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2145.253129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2145.261753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.291771] LustreError: 65303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.449722] Lustre: server umount lustre-OST0000 complete [ 2150.881929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2150.889061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2150.904711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2153.443614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2153.447688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.560397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2158.570902] Lustre: Skipped 2 previous similar messages [ 2159.677105] Lustre: server umount lustre-MDT0000 complete [ 2163.243265] LustreError: 63461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379707 with bad export cookie 883896013733443730 [ 2163.246820] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2163.249040] LustreError: 63461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2163.526895] Lustre: server umount lustre-MDT0001 complete [ 2173.888511] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2176.318641] Key type lgssc unregistered [ 2176.686243] LNet: 67041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.701342] LNetError: 67041:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2176.718268] LNet: Removed LNI 192.168.203.116@tcp [ 2177.517608] Key type .llcrypt unregistered [ 2177.520164] Key type ._llcrypt unregistered [ 2199.735806] Key type ._llcrypt registered [ 2199.740966] Key type .llcrypt registered [ 2199.888308] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2200.831749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2200.867118] alg: No test for adler32 (adler32-zlib) [ 2201.951604] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 2202.198378] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2203.935177] Key type lgssc registered [ 2204.974236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2213.242934] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 02:22:37 (1787379757) [ 2218.885687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2218.909239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.319675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.435379] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2230.756496] LustreError: 68724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.771467] LustreError: 68724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2232.203254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.578839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2236.588707] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2243.191312] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2248.437874] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2253.908458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2254.265808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.144816] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2265.516095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2265.842129] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2272.220833] Lustre: Failing over lustre-MDT0000 [ 2272.621136] Lustre: server umount lustre-MDT0000 complete [ 2273.766318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2273.774782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.778636] LustreError: 68720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.811936] LustreError: 68720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2275.808972] LustreError: 69452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.812084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.827917] LustreError: 69452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2275.838894] Lustre: Skipped 1 previous similar message [ 2278.874349] LustreError: 70564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.130139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2281.368555] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2281.518199] Lustre: *** cfs_fail_loc=135, val=0*** [ 2281.605350] LustreError: 71810:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2281.616319] LustreError: 71810:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.116@tcp: cfg command failed: rc = -2 [ 2281.632700] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2281.632700] [ 2281.641854] LustreError: MGC192.168.203.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2281.658610] LustreError: 71798:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2281.666208] LustreError: 71798:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2281.676910] LustreError: 71798:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2281.680078] LustreError: 71798:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2281.704589] Lustre: server umount lustre-MDT0000 complete [ 2281.708590] LustreError: 71798:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2283.999518] LustreError: 68720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.020898] LustreError: 68720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2288.241670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.414231] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2288.707727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.775520] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2289.176695] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2292.721454] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2293.741680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2293.769860] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2293.831926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2303.458407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2303.472211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.497160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2305.885378] Lustre: server umount lustre-OST0000 complete [ 2309.090013] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.097276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.104389] Lustre: Skipped 2 previous similar messages [ 2309.116641] Lustre: Skipped 2 previous similar messages [ 2314.210460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2314.225099] Lustre: Skipped 1 previous similar message [ 2315.362925] Lustre: server umount lustre-MDT0000 complete [ 2319.335979] LustreError: 68725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.368725] LustreError: 68725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2319.455698] LustreError: 68707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379863 with bad export cookie 13032043153341526648 [ 2319.462935] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.470483] LustreError: 68707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2319.916745] Lustre: server umount lustre-MDT0001 complete [ 2331.211981] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2333.828604] Key type lgssc unregistered [ 2334.132696] LNet: 73588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.144965] LNetError: 73588:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.165226] LNet: Removed LNI 192.168.203.116@tcp [ 2335.054511] Key type .llcrypt unregistered [ 2335.057107] Key type ._llcrypt unregistered [ 2359.090310] Key type ._llcrypt registered [ 2359.092701] Key type .llcrypt registered [ 2359.218414] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2360.353300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2360.388497] alg: No test for adler32 (adler32-zlib) [ 2361.405728] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 2361.615630] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2363.263164] Key type lgssc registered [ 2364.459031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2371.009226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2371.032036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.461167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.512767] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2382.829963] LustreError: 74999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.863304] LustreError: 74999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2385.353051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.878390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2391.025114] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2398.533794] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2403.323501] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2410.911065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2411.223646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.819665] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2417.396795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2425.331945] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2434.444484] Lustre: Failing over lustre-MDT0000 [ 2434.749977] Lustre: server umount lustre-MDT0000 complete [ 2436.067334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.071703] LustreError: 74995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.096445] Lustre: Skipped 2 previous similar messages [ 2436.107712] LustreError: 74995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2437.621688] LustreError: 74994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.194187] LustreError: 74994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.226099] LustreError: 74994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2444.070434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.193090] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2444.298793] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2444.399563] LustreError: 78084:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2444.407831] LustreError: 78084:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.116@tcp: cfg command failed: rc = -9 [ 2444.421330] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2444.421330] [ 2444.440038] LustreError: MGC192.168.203.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2444.460813] LustreError: 78073:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2444.471211] LustreError: 78073:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2444.490390] LustreError: 78073:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2444.508520] LustreError: 78073:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2444.534197] Lustre: server umount lustre-MDT0000 complete [ 2444.542180] LustreError: 78073:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2446.306263] LustreError: 74995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.328823] LustreError: 74995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2450.352976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.478750] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2450.708864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.794710] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2452.954833] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2456.060747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2456.120984] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2456.180891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2456.860917] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2466.284144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.290128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2466.303903] Lustre: Skipped 1 previous similar message [ 2466.312841] Lustre: Skipped 1 previous similar message [ 2471.468562] Lustre: server umount lustre-OST0000 complete [ 2476.515903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2476.516707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.518152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.553881] Lustre: Skipped 1 previous similar message [ 2481.632128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2481.639090] Lustre: Skipped 2 previous similar messages [ 2482.028892] Lustre: server umount lustre-MDT0000 complete [ 2485.863016] LustreError: 74978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380029 with bad export cookie 16996593641903654100 [ 2485.870333] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2485.890417] LustreError: 74978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2486.299480] Lustre: server umount lustre-MDT0001 complete [ 2498.240017] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2501.278698] Key type lgssc unregistered [ 2501.591034] LNet: 79867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2501.607912] LNetError: 79867:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2501.640304] LNet: Removed LNI 192.168.203.116@tcp [ 2502.714667] Key type .llcrypt unregistered [ 2502.716261] Key type ._llcrypt unregistered [ 2516.235499] Key type ._llcrypt registered [ 2516.239422] Key type .llcrypt registered [ 2516.374325] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2516.643835] Key type .llcrypt unregistered [ 2516.646306] Key type ._llcrypt unregistered [ 2539.627213] Key type ._llcrypt registered [ 2539.631492] Key type .llcrypt registered [ 2539.778260] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2541.052801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2541.083869] alg: No test for adler32 (adler32-zlib) [ 2542.114313] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 2542.315769] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2543.983203] Key type lgssc registered [ 2545.083521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2551.749848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2551.777067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.269288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.789871] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2563.558664] LustreError: 81771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.583493] LustreError: 81771:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2566.022661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.348383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2570.776789] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2578.634891] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2584.176780] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2591.347250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2591.793336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2597.450556] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2598.969194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2604.639878] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2619.361107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2619.368959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.391651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2621.408193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2621.422429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2621.423508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.447903] Lustre: Skipped 1 previous similar message [ 2624.071858] Lustre: server umount lustre-OST0000 complete [ 2628.065315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2628.080791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2628.092333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2631.652042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.652988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2631.675856] Lustre: Skipped 1 previous similar message [ 2636.770502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2636.774271] Lustre: Skipped 1 previous similar message [ 2642.399118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2642.562628] Lustre: server umount lustre-MDT0000 complete [ 2646.185862] LustreError: 81752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380190 with bad export cookie 10113349601917075643 [ 2646.188327] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.195714] LustreError: 81752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2646.670059] Lustre: server umount lustre-MDT0001 complete [ 2657.867736] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2660.292752] Key type lgssc unregistered [ 2660.595846] LNet: 85335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2660.609849] LNetError: 85335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2660.629848] LNet: Removed LNI 192.168.203.116@tcp [ 2661.568906] Key type .llcrypt unregistered [ 2661.571539] Key type ._llcrypt unregistered [ 2687.582489] Key type ._llcrypt registered [ 2687.584307] Key type .llcrypt registered [ 2687.705619] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2688.510891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2688.540346] alg: No test for adler32 (adler32-zlib) [ 2689.519635] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 2689.776973] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2691.391152] Key type lgssc registered [ 2692.352236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2698.765712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2698.792496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.243690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.795096] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2710.497984] LustreError: 86749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.516817] LustreError: 86749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2713.442761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.907247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2717.857550] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2723.941937] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2730.698185] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2739.428772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2739.961973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.084170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2747.131956] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2754.636693] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2765.792383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2765.808185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.821480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2768.865690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.878800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2768.892836] Lustre: Skipped 1 previous similar message [ 2771.668987] Lustre: server umount lustre-OST0000 complete [ 2779.106106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2779.110328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2779.116050] Lustre: Skipped 1 previous similar message [ 2779.130743] Lustre: Skipped 1 previous similar message [ 2784.224248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2784.231812] Lustre: Skipped 1 previous similar message [ 2789.351753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2789.365212] Lustre: Skipped 1 previous similar message [ 2790.367181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2790.614478] Lustre: server umount lustre-MDT0000 complete [ 2794.170698] LustreError: 86729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380338 with bad export cookie 13112083449467311044 [ 2794.174830] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2794.188744] LustreError: 86729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2794.972458] Lustre: server umount lustre-MDT0001 complete [ 2805.507611] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2808.576996] Key type lgssc unregistered [ 2808.906688] LNet: 90284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2808.929735] LNetError: 90284:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2808.949571] LNet: Removed LNI 192.168.203.116@tcp [ 2809.799729] Key type .llcrypt unregistered [ 2809.804935] Key type ._llcrypt unregistered [ 2834.341253] Key type ._llcrypt registered [ 2834.345797] Key type .llcrypt registered [ 2834.456660] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2835.818868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2835.941414] alg: No test for adler32 (adler32-zlib) [ 2837.111910] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 2837.366493] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2839.039175] Key type lgssc registered [ 2840.421784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2848.945993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2848.958219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.559649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.217522] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2860.002935] LustreError: 91698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.021057] LustreError: 91698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2865.122260] LustreError: 91699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.390281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.812820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2869.295905] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2877.406744] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2883.312800] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2889.855556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2890.241371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2895.345044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2896.110621] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2903.178937] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2916.833642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2916.846270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.863225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2920.928669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2920.933549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2920.940715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.550654] Lustre: server umount lustre-OST0000 complete [ 2927.075793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2927.084378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2927.101870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2927.109566] Lustre: Skipped 1 previous similar message [ 2931.171599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2931.174451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.974504] Lustre: server umount lustre-MDT0000 complete [ 2936.291162] LustreError: 91695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.312856] LustreError: 91695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2936.618279] LustreError: 91679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380480 with bad export cookie 10751295107050321694 [ 2936.619263] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.634361] LustreError: 91679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2937.024890] Lustre: server umount lustre-MDT0001 complete [ 2947.549229] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2950.238495] Key type lgssc unregistered [ 2950.460430] LNet: 95251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2950.478745] LNetError: 95251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2950.494584] LNet: Removed LNI 192.168.203.116@tcp [ 2951.255206] Key type .llcrypt unregistered [ 2951.258320] Key type ._llcrypt unregistered [ 2973.415272] Key type ._llcrypt registered [ 2973.417448] Key type .llcrypt registered [ 2973.508518] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2974.595523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2974.739802] alg: No test for adler32 (adler32-zlib) [ 2975.800638] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 2976.060955] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2977.695766] Key type lgssc registered [ 2978.680730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2985.455908] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2985.473794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3000.991482] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3006.111321] LustreError: 96630:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 3032.799802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.019552] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3046.657783] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3052.787850] Lustre: server umount lustre-OST0000 complete [ 3062.179687] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:36:45 (1787380605) [ 3068.641826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.132042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.069880] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3074.534183] LustreError: 98267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.564320] LustreError: 98267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3079.657120] LustreError: 98268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3081.730495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.131207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3086.444671] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3094.695370] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3099.921079] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3106.605490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3106.880407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3112.679385] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3114.048053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3120.476620] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3133.413226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3133.430719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.445067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3134.436939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3134.440694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3134.446496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.992896] Lustre: server umount lustre-OST0000 complete [ 3143.648056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3143.654710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.665791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3143.671188] Lustre: Skipped 1 previous similar message [ 3148.452589] Lustre: server umount lustre-MDT0000 complete [ 3149.798260] LustreError: 101009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.826261] LustreError: 101009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3151.998047] LustreError: 99002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380696 with bad export cookie 12446701368893694059 [ 3152.011337] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3152.014097] LustreError: 99002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3152.334266] Lustre: server umount lustre-MDT0001 complete [ 3163.962289] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3167.417697] Key type lgssc unregistered [ 3167.763637] LNet: 101962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3167.775758] LNetError: 101962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3167.792727] LNet: Removed LNI 192.168.203.116@tcp [ 3168.513164] Key type .llcrypt unregistered [ 3168.520631] Key type ._llcrypt unregistered [ 3211.938486] Key type ._llcrypt registered [ 3211.946352] Key type .llcrypt registered [ 3212.068871] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3213.412828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3213.487709] alg: No test for adler32 (adler32-zlib) [ 3214.551456] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 3214.803336] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3216.495228] Key type lgssc registered [ 3217.540703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3225.204683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3225.237400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3226.776092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3231.009241] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3236.321594] LustreError: 104150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3236.366089] LustreError: 104150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3239.380516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.890855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3244.791068] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3252.492129] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3258.866828] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3265.934431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.240195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3271.359824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3273.312938] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3280.727433] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3294.188787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3294.200451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.218813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3295.202891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3295.205159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3295.215245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3295.251463] Lustre: Skipped 1 previous similar message [ 3298.975341] Lustre: server umount lustre-OST0000 complete [ 3305.445351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.463765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3305.469099] Lustre: Skipped 1 previous similar message [ 3308.828578] Lustre: server umount lustre-MDT0000 complete [ 3310.563598] LustreError: 104149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3310.590703] LustreError: 104149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3312.516960] LustreError: 104132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380856 with bad export cookie 16351920214086499858 [ 3312.519803] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3312.532932] LustreError: 104132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3312.878697] Lustre: server umount lustre-MDT0001 complete [ 3324.077521] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3326.768740] Key type lgssc unregistered [ 3327.092889] LNet: 107697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3327.100037] LNetError: 107697:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3327.127324] LNet: Removed LNI 192.168.203.116@tcp [ 3328.014434] Key type .llcrypt unregistered [ 3328.017096] Key type ._llcrypt unregistered [ 3345.378211] Key type ._llcrypt registered [ 3345.381222] Key type .llcrypt registered [ 3345.503902] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3346.715881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3346.883786] alg: No test for adler32 (adler32-zlib) [ 3348.040495] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 3348.335919] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3350.079192] Key type lgssc registered [ 3351.420688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3360.624876] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3360.659939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3376.287934] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3381.407413] LustreError: 108932:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 3408.095324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.292281] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3424.228092] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3431.870504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.096085] LustreError: MGC192.168.203.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3432.105313] LustreError: 109958:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3432.110910] LustreError: 109958:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3432.120287] LustreError: 109958:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3432.209469] Lustre: server umount lustre-MDT0000 complete [ 3432.213704] LustreError: 109958:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3452.383293] Lustre: 108269:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787380980/real 1787380980] req@00000000a2f68814 x1874204706213248/t0(0) o400->MGC192.168.203.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787380996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3452.425295] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3452.440765] LustreError: 108945:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3473.032103] Lustre: server umount lustre-OST0000 complete [ 3482.057911] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3491.798647] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3531.281387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.533355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3531.568316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3531.673749] Lustre: lustre-MDT0000: new disk, initializing [ 3531.784303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3531.813390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3537.118522] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3547.575962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3547.732994] Lustre: 112752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3547.783654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3547.788874] Lustre: Skipped 1 previous similar message [ 3547.855526] Lustre: lustre-MDT0001: new disk, initializing [ 3548.000973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3548.051068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3548.067740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3552.595372] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3561.238624] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3566.996502] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3573.612682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3573.937186] Lustre: lustre-OST0000: new disk, initializing [ 3573.940397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3573.945741] Lustre: 114723:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3574.027310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.158261] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3583.054612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3583.066190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3583.164384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3589.072388] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3593.668753] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3593.927201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3598.662095] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3598.853172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3603.939123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3603.942869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3603.946637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3603.952988] LustreError: Skipped 1 previous similar message [ 3603.999176] Lustre: Skipped 1 previous similar message [ 3607.041336] Lustre: server umount lustre-OST0000 complete [ 3614.179046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3614.198811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3614.207859] Lustre: Skipped 2 previous similar messages [ 3619.425589] Lustre: server umount lustre-MDT0000 complete [ 3623.185393] LustreError: 112744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381167 with bad export cookie 1997893458468879914 [ 3623.188022] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3623.197133] LustreError: 112744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3623.674484] Lustre: server umount lustre-MDT0001 complete [ 3633.128277] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:46:16 (1787381176) [ 3636.536434] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3651.636510] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3661.674413] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3696.809218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3696.999295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3697.037073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3697.129169] Lustre: lustre-MDT0000: new disk, initializing [ 3697.199196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3697.218987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3701.602738] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3712.203247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3712.331176] Lustre: 119619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3712.379293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3712.383150] Lustre: Skipped 1 previous similar message [ 3712.456431] Lustre: lustre-MDT0001: new disk, initializing [ 3712.557101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3712.622266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3712.649606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3717.269983] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3728.027143] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3733.906333] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3740.707197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3740.990759] Lustre: lustre-OST0000: new disk, initializing [ 3740.993994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3741.000977] Lustre: 121587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3741.101292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3747.107386] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3749.451718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3749.470143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3749.575614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3756.999579] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3762.040858] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3762.277431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3767.281289] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3767.629634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3770.348078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3770.350435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3770.371043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3770.406234] Lustre: Skipped 2 previous similar messages [ 3775.462833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3775.474557] Lustre: Skipped 2 previous similar messages [ 3776.287769] Lustre: server umount lustre-OST0000 complete [ 3780.579685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.596303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3780.604409] Lustre: Skipped 1 previous similar message [ 3790.816767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3790.826730] Lustre: Skipped 3 previous similar messages [ 3794.399084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3794.619792] Lustre: server umount lustre-MDT0000 complete [ 3795.939261] LustreError: 120496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3795.971554] LustreError: 120496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3798.499323] LustreError: 119612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381342 with bad export cookie 1997893458468880796 [ 3798.505246] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3798.514416] LustreError: 119612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3799.063737] Lustre: server umount lustre-MDT0001 complete [ 3808.668186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.213792] LustreError: 123809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3809.303776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.373163] LustreError: 123810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3814.458385] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3819.489371] LustreError: 123809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3824.194083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.288219] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3837.293776] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3843.942474] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3851.295773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3858.194799] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3866.504959] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3876.345673] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3884.514139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3884.520685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3884.527989] Lustre: Skipped 2 previous similar messages [ 3884.542996] Lustre: Skipped 1 previous similar message [ 3886.856510] Lustre: server umount lustre-OST0000 complete [ 3891.168346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3891.178917] LustreError: Skipped 1 previous similar message [ 3894.757550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.767087] Lustre: Skipped 1 previous similar message [ 3896.461514] Lustre: server umount lustre-MDT0000 complete [ 3899.872629] LustreError: 126107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.894459] LustreError: 126107:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3899.939384] LustreError: 124545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381444 with bad export cookie 1997893458468881678 [ 3899.942433] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3899.957948] LustreError: 124545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3900.415900] Lustre: server umount lustre-MDT0001 complete [ 3910.547961] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3913.005825] Key type lgssc unregistered [ 3913.331198] LNet: 127441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3913.345321] LNetError: 127441:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3913.362199] LNet: Removed LNI 192.168.203.116@tcp [ 3914.352884] Key type .llcrypt unregistered [ 3914.358206] Key type ._llcrypt unregistered [ 3932.281314] Key type ._llcrypt registered [ 3932.285450] Key type .llcrypt registered [ 3932.382218] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3948.380423] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3949.749398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3949.851943] alg: No test for adler32 (adler32-zlib) [ 3951.015239] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 3951.341843] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3953.031434] Key type lgssc registered [ 3954.441770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3991.447289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3991.490838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3992.792384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3992.864660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3992.968865] Lustre: lustre-MDT0000: new disk, initializing [ 3993.068280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3993.083364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3999.515851] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4011.258864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.453985] Lustre: 130600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4011.589994] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4011.598791] Lustre: Skipped 1 previous similar message [ 4011.748644] Lustre: lustre-MDT0001: new disk, initializing [ 4011.886958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4011.989821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4012.003670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4016.115242] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4024.737955] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4030.524611] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4036.957446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4037.281497] Lustre: lustre-OST0000: new disk, initializing [ 4037.284597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4037.297737] Lustre: 132567:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4037.391650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.580993] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4043.878941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4043.896519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4044.019992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4052.721787] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4057.265723] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4057.535505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4061.846803] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4062.036356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4064.747338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.749525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4064.766565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4064.773380] LustreError: Skipped 1 previous similar message [ 4064.782712] Lustre: Skipped 1 previous similar message [ 4069.866814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4069.872792] Lustre: Skipped 1 previous similar message [ 4070.098841] Lustre: server umount lustre-OST0000 complete [ 4074.979078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4074.983735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4074.990514] Lustre: Skipped 2 previous similar messages [ 4075.007866] Lustre: Skipped 1 previous similar message [ 4080.098663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4080.108773] Lustre: Skipped 1 previous similar message [ 4080.398658] Lustre: server umount lustre-MDT0000 complete [ 4084.079611] LustreError: 130590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381628 with bad export cookie 1098373181004353671 [ 4084.094252] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4084.098280] LustreError: 130590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4084.375427] Lustre: server umount lustre-MDT0001 complete [ 4093.297935] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:53:57 (1787381637) [ 4099.920139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.608505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.530916] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4105.702249] LustreError: 134980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4105.718907] LustreError: 134980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4110.823175] LustreError: 134979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4112.706067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.207970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4117.595623] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4124.740393] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4130.090940] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4131.926845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4131.932965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4131.955901] Lustre: Skipped 1 previous similar message [ 4131.969548] Lustre: Skipped 1 previous similar message [ 4138.313804] Lustre: server umount lustre-MDT0000 complete [ 4141.646379] LustreError: 134959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381685 with bad export cookie 1098373181004354546 [ 4141.655945] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4141.658588] LustreError: 134959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4141.948375] Lustre: server umount lustre-MDT0001 complete [ 4150.840584] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:54:54 (1787381694) [ 4156.897941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4172.511634] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4177.632876] LustreError: 137394:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 4203.423288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4209.679898] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4218.297408] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4220.543322] Lustre: server umount lustre-OST0000 complete [ 4229.188774] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:56:13 (1787381773) [ 4235.734989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.226892] LustreError: 138839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.304566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4240.720879] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4241.376485] LustreError: 138840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4246.498419] LustreError: 138839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4248.823096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4253.452927] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4260.618409] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4266.304414] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4272.002102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4272.468296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4272.479368] Lustre: Skipped 1 previous similar message [ 4278.816712] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4287.424798] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4297.185927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4297.189229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4297.225327] Lustre: Skipped 2 previous similar messages [ 4297.243205] Lustre: Skipped 6 previous similar messages [ 4302.849918] Lustre: server umount lustre-MDT0000 complete [ 4307.143121] LustreError: 138821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381851 with bad export cookie 1098373181004355218 [ 4307.155133] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4307.156567] LustreError: 138821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4307.455447] Lustre: server umount lustre-MDT0001 complete [ 4321.920759] Lustre: server umount lustre-OST0000 complete [ 4331.034900] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:57:54 (1787381874) [ 4337.839813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4338.340497] LustreError: 142492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4338.434107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4343.116669] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4351.690703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4356.387337] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4364.103966] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4369.804565] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4377.102987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4383.670623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4384.076358] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4392.789553] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4397.666473] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4397.909521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4403.204582] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4403.423581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4407.268923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.300789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4407.303631] Lustre: Skipped 3 previous similar messages [ 4411.873632] Lustre: server umount lustre-OST0000 complete [ 4417.510519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.517548] Lustre: Skipped 1 previous similar message [ 4422.638813] LustreError: 143735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4422.658950] LustreError: 143735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4422.748912] Lustre: server umount lustre-MDT0000 complete [ 4427.676241] LustreError: 142474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381971 with bad export cookie 1098373181004356289 [ 4427.685180] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4427.689670] LustreError: 142474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4428.277985] Lustre: server umount lustre-MDT0001 complete [ 4437.863505] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:59:41 (1787381981) [ 4444.892098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4460.386823] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4465.505935] LustreError: 146654:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 4491.231502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4491.234324] Lustre: Skipped 2 previous similar messages [ 4496.963489] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4505.218900] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4513.231325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4513.764613] LustreError: 146679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4513.792238] LustreError: 146679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4513.924162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4519.152673] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4528.189754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.255748] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4542.225883] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4548.343246] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4555.138399] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4555.494097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4561.911503] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4562.188293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4569.570493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4569.572927] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4569.591225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4569.600579] Lustre: Skipped 2 previous similar messages [ 4569.625744] Lustre: Skipped 4 previous similar messages [ 4579.295133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4579.534667] Lustre: server umount lustre-MDT0000 complete [ 4579.810444] LustreError: 147691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.828586] LustreError: 147691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4583.519174] LustreError: 146662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382127 with bad export cookie 1098373181004357311 [ 4583.529618] LustreError: 146662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4583.535563] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.807741] Lustre: server umount lustre-MDT0001 complete [ 4598.483395] Lustre: server umount lustre-OST0000 complete [ 4606.615506] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 03:02:30 (1787382150) [ 4613.277564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4628.895776] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4634.015469] LustreError: 150864:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 4665.694424] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4673.314849] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4682.369868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.967267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4687.388688] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4697.369681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.708400] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4710.624313] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4716.751655] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4723.816792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4723.944489] Lustre: lustre-OST0001: new disk, initializing [ 4723.949677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4723.956195] Lustre: 153738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4730.346663] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4731.428172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4731.438129] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4731.545752] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4739.622979] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4743.826956] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4744.006645] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4747.831746] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4748.021449] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4749.799331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4749.809711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4749.823859] Lustre: Skipped 1 previous similar message [ 4749.831354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4749.838958] Lustre: Skipped 3 previous similar messages [ 4764.127394] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4764.234566] Lustre: server umount lustre-OST0000 complete [ 4767.212697] LustreError: 154151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4767.253634] LustreError: 154151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4772.320942] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4772.334795] LustreError: Skipped 1 previous similar message [ 4775.520582] Lustre: server umount lustre-OST0001 complete [ 4786.806653] Lustre: server umount lustre-MDT0000 complete [ 4790.314615] LustreError: 150870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382334 with bad export cookie 1098373181004358263 [ 4790.321663] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4790.330971] LustreError: 150870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4790.748039] Lustre: server umount lustre-MDT0001 complete [ 4813.058746] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 03:05:57 (1787382357) [ 4814.821857] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4816.163822] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4817.611631] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 03:06:01 (1787382361) [ 4824.184729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4824.444830] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4824.498972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4824.703587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4824.709696] Lustre: Skipped 6 previous similar messages [ 4828.274113] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4835.343566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.416519] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4835.454392] Lustre: 157299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4839.988870] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4847.051613] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4853.511689] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4860.291890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4860.487272] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4866.202439] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4869.689993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4872.857862] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4878.326696] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4878.586256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4883.361345] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4883.555713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4890.096096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4890.096565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4890.104338] LustreError: Skipped 1 previous similar message [ 4890.116315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4890.116359] Lustre: Skipped 6 previous similar messages [ 4890.136122] Lustre: Skipped 11 previous similar messages [ 4891.414707] Lustre: server umount lustre-OST0000 complete [ 4915.030887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4922.547895] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4929.983432] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4929.997141] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4930.004601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.527682] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4935.092592] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4935.306568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4940.219881] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4940.470848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4945.876431] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff929bc8af0800.ost_server_uuid 50 [ 4947.778761] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff929bc8af0800.ost_server_uuid in FULL state after 0 sec [ 4947.986533] LustreError: 159496:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.16@tcp) returned error from blocking AST (req@00000000f2a43aa8 x1874205338766080 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000a2f960f4/0xf3e3496b494abd1 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.16@tcp remote: 0x11dbd947cc799967 expref: 5 pid: 157305 timeout: 5047 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4948.021807] LustreError: lustre-MDT0000: A client on nid 192.168.203.16@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4948.029199] LustreError: 157294:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.203.16@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000a2f960f4/0xf3e3496b494abd1 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.16@tcp remote: 0x11dbd947cc799967 expref: 6 pid: 157305 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4954.083181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4959.505245] Lustre: server umount lustre-OST0000 complete [ 4970.088603] Lustre: server umount lustre-MDT0000 complete [ 4974.040106] LustreError: 157289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382518 with bad export cookie 1098373181004359446 [ 4974.047548] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4974.062405] LustreError: 157289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4988.383666] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4988.861700] Lustre: server umount lustre-MDT0001 complete [ 5000.030903] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5002.892378] Key type lgssc unregistered [ 5003.156797] LNet: 163417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5003.169893] LNetError: 163417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5003.184689] LNet: Removed LNI 192.168.203.116@tcp [ 5004.028439] Key type .llcrypt unregistered [ 5004.032125] Key type ._llcrypt unregistered [ 5028.420317] Key type ._llcrypt registered [ 5028.423802] Key type .llcrypt registered [ 5028.511793] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5029.512520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5029.634790] alg: No test for adler32 (adler32-zlib) [ 5030.730890] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 5030.951184] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5032.680092] Key type lgssc registered [ 5033.841550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5041.201920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5041.230045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5042.657312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5047.205209] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5052.925539] LustreError: 164832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5052.954295] LustreError: 164832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5056.838666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.269706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5061.567664] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5069.314860] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5075.697798] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5083.184403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5083.507034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5087.617081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5089.631949] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5098.192303] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5108.047396] Lustre: Failing over lustre-MDT0000 [ 5108.198479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5108.207228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5108.213063] Lustre: Skipped 2 previous similar messages [ 5108.228792] Lustre: Skipped 2 previous similar messages [ 5108.349437] Lustre: server umount lustre-MDT0000 complete [ 5113.313907] LustreError: 164827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.344148] LustreError: 164827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5118.436658] LustreError: 164832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.447917] LustreError: 164832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5120.746405] LustreError: 164812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.16@tcp arrived at 1787382664 with bad export cookie 12823454197499501946 [ 5120.772116] LustreError: 164812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5123.554342] LustreError: 164833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5123.578507] LustreError: 164833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5127.178565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5127.271672] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5127.452850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5127.497337] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5130.975600] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5132.750119] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5132.756270] Lustre: lustre-MDT0000: Denying connection for new client fce6ed3c-6718-4fc3-a95e-b7d73cf9476a (at 192.168.203.16@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5132.790238] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5143.012750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5143.017513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5143.026183] Lustre: Skipped 1 previous similar message [ 5143.035460] Lustre: Skipped 1 previous similar message [ 5146.767660] Lustre: server umount lustre-OST0000 complete [ 5149.680564] LustreError: 168562:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5149.686033] Lustre: 167910:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5149.690617] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5149.700344] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5149.708655] LustreError: 167910:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c7d63598 x1874206471007104/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5149.724990] LustreError: 167910:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5149.744816] LustreError: 167910:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5149.752682] Lustre: lustre-MDT0000: Recovery over after 0:17, of 2 clients 0 recovered and 2 were evicted. [ 5150.003271] Lustre: server umount lustre-MDT0000 complete [ 5153.054881] LustreError: 167527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382697 with bad export cookie 12823454197499502051 [ 5153.060723] LustreError: 164140:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5153.064053] LustreError: 167527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5153.068668] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5155.445933] Lustre: server umount lustre-MDT0001 complete [ 5164.734201] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5167.736959] Key type lgssc unregistered [ 5168.062253] LNet: 169312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5168.069312] LNetError: 169312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5168.090139] LNet: Removed LNI 192.168.203.116@tcp [ 5169.035771] Key type .llcrypt unregistered [ 5169.038846] Key type ._llcrypt unregistered [ 5193.812835] Key type ._llcrypt registered [ 5193.815388] Key type .llcrypt registered [ 5193.897497] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5194.963576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5195.007937] alg: No test for adler32 (adler32-zlib) [ 5196.024069] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 5196.282971] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5197.947179] Key type lgssc registered [ 5199.105446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5206.473834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5206.498265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5207.895834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5212.033334] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5217.250357] LustreError: 170723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.279549] LustreError: 170723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5220.708551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5221.173256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5225.347541] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5232.896981] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5238.546695] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5245.096440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5245.467233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5251.718795] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5252.649812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5258.536236] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5266.919862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5266.927578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.948939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5267.570134] Lustre: server umount lustre-OST0000 complete [ 5272.544680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5272.554734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5272.565231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5278.176795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5278.186650] Lustre: Skipped 3 previous similar messages [ 5283.300940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5283.309410] Lustre: Skipped 1 previous similar message [ 5285.345260] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5285.524572] Lustre: server umount lustre-MDT0000 complete [ 5288.417667] LustreError: 171977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.426657] LustreError: 171977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5289.231920] LustreError: 170705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382833 with bad export cookie 7036932296456291726 [ 5289.239374] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5289.246973] LustreError: 170705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5289.526818] Lustre: server umount lustre-MDT0001 complete [ 5301.022960] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5304.069504] Key type lgssc unregistered [ 5304.390285] LNet: 174257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5304.406212] LNetError: 174257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5304.429680] LNet: Removed LNI 192.168.203.116@tcp [ 5305.494582] Key type .llcrypt unregistered [ 5305.500612] Key type ._llcrypt unregistered [ 5340.875360] Key type ._llcrypt registered [ 5340.878062] Key type .llcrypt registered [ 5340.976884] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5342.209744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5342.390430] alg: No test for adler32 (adler32-zlib) [ 5343.472365] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 5343.765211] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5345.503336] Key type lgssc registered [ 5346.593530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5353.484829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5353.506894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5354.882463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5358.641274] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5364.200185] LustreError: 175960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5364.215238] LustreError: 175960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5367.661098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.136614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5372.655981] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5379.752385] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5386.675972] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5393.706795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5393.835825] LustreError: 177783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5394.124331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.285606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5400.009617] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5406.760967] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5414.879988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5414.886367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5414.898844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5414.903342] Lustre: Skipped 1 previous similar message [ 5414.908503] Lustre: Skipped 1 previous similar message [ 5416.299158] Lustre: server umount lustre-OST0000 complete [ 5423.077978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5423.084535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5423.091305] Lustre: Skipped 1 previous similar message [ 5423.111648] Lustre: Skipped 1 previous similar message [ 5425.120967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5425.130706] Lustre: Skipped 1 previous similar message [ 5426.915528] Lustre: server umount lustre-MDT0000 complete [ 5430.243826] LustreError: 177201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5430.264298] LustreError: 177201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5430.395899] LustreError: 175940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382974 with bad export cookie 6968257967046870064 [ 5430.417320] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5430.420471] LustreError: 175940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5430.760559] Lustre: server umount lustre-MDT0001 complete [ 5441.658768] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5444.183061] Key type lgssc unregistered [ 5444.463277] LNet: 179494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5444.471186] LNetError: 179494:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5444.483227] LNet: Removed LNI 192.168.203.116@tcp [ 5445.282895] Key type .llcrypt unregistered [ 5445.285957] Key type ._llcrypt unregistered [ 5465.644161] Key type ._llcrypt registered [ 5465.648270] Key type .llcrypt registered [ 5465.773406] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5466.399709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5466.431089] alg: No test for adler32 (adler32-zlib) [ 5467.538379] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 5467.746695] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5469.392344] Key type lgssc registered [ 5470.666925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5480.603243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5480.638397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5482.034205] Lustre: *** cfs_fail_loc=135, val=0*** [ 5482.088449] LustreError: 181017:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5482.093292] LustreError: 181017:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.116@tcp: cfg command failed: rc = -2 [ 5482.101305] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5482.101305] [ 5482.110503] LustreError: MGC192.168.203.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5482.122979] LustreError: 180978:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5482.132479] LustreError: 180978:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5482.139710] LustreError: 180978:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5482.152162] LustreError: 180978:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5482.233869] Lustre: server umount lustre-MDT0000 complete [ 5482.236400] LustreError: 180978:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5488.480077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5488.831376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5492.997085] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5498.343332] LustreError: 181389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5498.363756] LustreError: 181389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5500.872536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5501.168052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5505.828892] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5513.677974] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5519.771827] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5521.775257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5521.793202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5521.801061] Lustre: Skipped 1 previous similar message [ 5526.948120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5526.962891] Lustre: Skipped 1 previous similar message [ 5532.130048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5532.137550] Lustre: Skipped 1 previous similar message [ 5536.223548] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5536.443645] Lustre: server umount lustre-MDT0000 complete [ 5537.250417] LustreError: 182401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.262807] LustreError: 182401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5540.238498] LustreError: 181371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383084 with bad export cookie 1208466931856175169 [ 5540.245130] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5540.249807] LustreError: 181371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5540.696782] Lustre: server umount lustre-MDT0001 complete [ 5551.055726] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5553.459630] Key type lgssc unregistered [ 5553.754536] LNet: 183710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5553.760424] LNetError: 183710:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5553.778011] LNet: Removed LNI 192.168.203.116@tcp [ 5554.618892] Key type .llcrypt unregistered [ 5554.620778] Key type ._llcrypt unregistered [ 5575.001106] Key type ._llcrypt registered [ 5575.006333] Key type .llcrypt registered [ 5575.111520] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5575.453360] Key type .llcrypt unregistered [ 5575.455684] Key type ._llcrypt unregistered [ 5590.121863] Key type ._llcrypt registered [ 5590.129047] Key type .llcrypt registered [ 5590.244778] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5591.910043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5591.959067] alg: No test for adler32 (adler32-zlib) [ 5593.173910] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 5593.482079] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5595.139089] Key type lgssc registered [ 5596.870391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5605.413109] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5605.454122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5621.088214] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5626.209749] LustreError: 185824:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 5652.896069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5659.000664] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5667.685827] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5674.739886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.849040] LustreError: 185158:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c687d0f5 x1874207060265216/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5675.188415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.248117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5679.826820] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5680.609444] LustreError: 185849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5680.661604] LustreError: 185849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5685.740543] LustreError: 185850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.787367] LustreError: 185849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.802262] LustreError: 185849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5688.372535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.930393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5693.615671] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5701.447658] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5707.012260] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5709.739404] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5721.569171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5721.575409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.603460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5724.641687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.643251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5726.606782] Lustre: server umount lustre-OST0000 complete [ 5734.880464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.889361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5734.917187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5734.917194] Lustre: Skipped 1 previous similar message [ 5736.608017] Lustre: server umount lustre-MDT0000 complete [ 5740.001375] LustreError: 186865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5740.027168] LustreError: 186865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5740.072571] LustreError: 185832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383284 with bad export cookie 11124537156731837647 [ 5740.083148] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5740.085923] LustreError: 185832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5740.376494] Lustre: server umount lustre-MDT0001 complete [ 5751.263264] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5753.859313] Key type lgssc unregistered [ 5754.143087] LNet: 189789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5754.162374] LNetError: 189789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5754.188106] LNet: Removed LNI 192.168.203.116@tcp [ 5754.895221] Key type .llcrypt unregistered [ 5754.900609] Key type ._llcrypt unregistered [ 5776.695434] Key type ._llcrypt registered [ 5776.697979] Key type .llcrypt registered [ 5776.781479] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5777.592508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5777.681734] alg: No test for adler32 (adler32-zlib) [ 5778.817540] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 5779.168745] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5780.879224] Key type lgssc registered [ 5782.053795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5790.381750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5790.415254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5791.965839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5796.885833] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5802.464586] LustreError: 191202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5802.489506] LustreError: 191202:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5807.589851] LustreError: 191201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5807.675883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.035758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5812.838129] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5820.625572] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5827.173864] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5833.972263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5834.207532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5837.296652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5840.655519] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5848.388830] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5853.681522] Lustre: Failing over lustre-MDT0000 [ 5853.988650] Lustre: server umount lustre-MDT0000 complete [ 5854.177065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5854.192773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.206705] LustreError: 191201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.231544] LustreError: 191198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5857.763493] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5857.773446] Lustre: Skipped 1 previous similar message [ 5861.336897] LustreError: 191196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.364699] LustreError: 191196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5871.570130] LustreError: 191198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.601500] LustreError: 191198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5872.373914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5872.488426] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5872.749991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5872.817193] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5876.693638] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5876.924518] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5878.251499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5878.278661] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5878.305327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5881.644839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5887.399709] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5898.726659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5898.728091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5898.731184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5898.735112] Lustre: Skipped 1 previous similar message [ 5898.757275] Lustre: Skipped 1 previous similar message [ 5900.397870] Lustre: server umount lustre-OST0000 complete [ 5908.960379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5908.965995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5908.969498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5909.000288] Lustre: Skipped 1 previous similar message [ 5910.444443] Lustre: server umount lustre-MDT0000 complete [ 5913.878651] LustreError: 191181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383457 with bad export cookie 4592678244325662960 [ 5913.880020] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5913.887532] LustreError: 191181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5914.225972] Lustre: server umount lustre-MDT0001 complete [ 5924.481510] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5927.520894] Key type lgssc unregistered [ 5927.867577] LNet: 196040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5927.874784] LNetError: 196040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5928.941341] LNet: Removed LNI 192.168.203.116@tcp [ 5929.721111] Key type .llcrypt unregistered [ 5929.725570] Key type ._llcrypt unregistered [ 5951.144492] Key type ._llcrypt registered [ 5951.148104] Key type .llcrypt registered [ 5951.249800] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5952.364832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5952.392385] alg: No test for adler32 (adler32-zlib) [ 5953.600650] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 5953.901901] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5955.559199] Key type lgssc registered [ 5956.875765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5965.280895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5965.309307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5966.886096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.547625] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5976.036150] LustreError: 197431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5976.075397] LustreError: 197431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5979.492165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.989545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5984.272729] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5991.912689] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5998.362059] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6006.967864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6007.466154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6011.601405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6015.912367] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6024.331628] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6028.615816] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6034.422861] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6052.757850] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6063.072913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6063.086550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6063.105461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6066.151912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6066.154843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6066.163483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.321525] Lustre: server umount lustre-OST0000 complete [ 6076.385039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.418482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6076.421479] Lustre: Skipped 1 previous similar message [ 6079.463936] Lustre: server umount lustre-MDT0000 complete [ 6081.505671] LustreError: 198158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6081.527869] LustreError: 198158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6082.910075] LustreError: 197411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383626 with bad export cookie 15626693645283521355 [ 6082.917250] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6082.922781] LustreError: 197411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6083.326173] Lustre: server umount lustre-MDT0001 complete [ 6093.823665] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6096.722296] Key type lgssc unregistered [ 6097.006127] LNet: 201298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6097.016131] LNetError: 201298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6097.043759] LNet: Removed LNI 192.168.203.116@tcp [ 6098.085707] Key type .llcrypt unregistered [ 6098.087641] Key type ._llcrypt unregistered [ 6116.219266] Key type ._llcrypt registered [ 6116.222097] Key type .llcrypt registered [ 6116.348297] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6117.523557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6117.544826] alg: No test for adler32 (adler32-zlib) [ 6118.507191] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 6118.721933] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6120.455942] Key type lgssc registered [ 6121.866343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6131.793907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6131.825675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.350800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6137.897358] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6142.958957] LustreError: 202564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.001424] LustreError: 202564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6145.890770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6146.315280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6150.651686] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6158.523528] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6163.623858] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6170.307108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6170.683654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6176.638869] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6178.880165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6182.882863] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6189.141761] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6201.153273] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6207.045633] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6214.266194] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6227.940794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6227.955811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6227.988706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6229.984939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6229.987215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6230.035412] Lustre: Skipped 1 previous similar message [ 6232.627701] Lustre: server umount lustre-OST0000 complete [ 6238.689539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6238.696650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6238.709231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6242.795379] Lustre: server umount lustre-MDT0000 complete [ 6245.344797] LustreError: 203293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6245.371052] LustreError: 203293:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6246.215174] LustreError: 205246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383790 with bad export cookie 3833157303080552825 [ 6246.217594] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6246.224320] LustreError: 205246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6246.599714] Lustre: server umount lustre-MDT0001 complete [ 6257.456328] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6260.060846] Key type lgssc unregistered [ 6260.429726] LNet: 207589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6260.441475] LNetError: 207589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6260.455907] LNet: Removed LNI 192.168.203.116@tcp [ 6261.377875] Key type .llcrypt unregistered [ 6261.380866] Key type ._llcrypt unregistered [ 6288.105300] Key type ._llcrypt registered [ 6288.109552] Key type .llcrypt registered [ 6288.300237] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6290.090369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6290.135190] alg: No test for adler32 (adler32-zlib) [ 6291.307518] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 6291.733214] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6293.455401] Key type lgssc registered [ 6294.824513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6302.298573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6302.348504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6303.754850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6308.157290] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6313.959586] LustreError: 209002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.979800] LustreError: 209002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6317.726660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.217585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6324.062657] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6332.922240] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6339.604345] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6346.255929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6346.652963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6349.746694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6353.021857] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6360.644914] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6375.392139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6375.404370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6375.423577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6378.466240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6378.474811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6378.476111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6378.497655] Lustre: Skipped 1 previous similar message [ 6379.768585] Lustre: server umount lustre-OST0000 complete [ 6383.585881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6383.594449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6383.595309] Lustre: Skipped 1 previous similar message [ 6383.606535] Lustre: Skipped 1 previous similar message [ 6388.705485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6388.710170] Lustre: Skipped 1 previous similar message [ 6389.550874] Lustre: server umount lustre-MDT0000 complete [ 6393.135888] LustreError: 208981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383937 with bad export cookie 13692969684866345011 [ 6393.144865] LustreError: 208981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6393.150193] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6393.406755] Lustre: server umount lustre-MDT0001 complete [ 6404.656822] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6407.396989] Key type lgssc unregistered [ 6407.669342] LNet: 212688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6407.684218] LNetError: 212688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6407.706277] LNet: Removed LNI 192.168.203.116@tcp [ 6408.624204] Key type .llcrypt unregistered [ 6408.628404] Key type ._llcrypt unregistered [ 6429.109668] Key type ._llcrypt registered [ 6429.114024] Key type .llcrypt registered [ 6429.230226] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6430.228608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6430.251922] alg: No test for adler32 (adler32-zlib) [ 6431.615898] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 6432.095291] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6433.791176] Key type lgssc registered [ 6435.348554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6443.146357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6443.155329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6444.571328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.460922] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6454.757179] LustreError: 213954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6454.778398] LustreError: 213954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6458.366388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6458.885248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6463.661955] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6471.282199] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6477.623453] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6484.877707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6485.292149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6489.418724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6491.669937] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6499.382152] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6515.169272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6515.181150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6515.200097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6519.268035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6519.280327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6519.282366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6519.332789] Lustre: Skipped 1 previous similar message [ 6520.656959] Lustre: server umount lustre-OST0000 complete [ 6525.417743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6525.425206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6525.440277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6529.504919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6529.505974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6534.626068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6534.642992] Lustre: Skipped 2 previous similar messages [ 6539.231314] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6539.448253] Lustre: server umount lustre-MDT0000 complete [ 6539.761937] LustreError: 216001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6539.798898] LustreError: 216001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6543.493929] LustreError: 214688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384087 with bad export cookie 13701135324916145374 [ 6543.499076] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6543.517437] LustreError: 214688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6543.915250] Lustre: server umount lustre-MDT0001 complete [ 6555.256344] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6558.125793] Key type lgssc unregistered [ 6558.395472] LNet: 217638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6558.411875] LNetError: 217638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6558.443054] LNet: Removed LNI 192.168.203.116@tcp [ 6559.574621] Key type .llcrypt unregistered [ 6559.578140] Key type ._llcrypt unregistered [ 6581.286860] Key type ._llcrypt registered [ 6581.290835] Key type .llcrypt registered [ 6581.429570] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6582.635101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6582.661270] alg: No test for adler32 (adler32-zlib) [ 6583.724572] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 6583.958869] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6585.647161] Key type lgssc registered [ 6586.691574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6596.290477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6596.308729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6597.929867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6603.563356] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6607.358407] LustreError: 218906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.382988] LustreError: 218906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6612.450946] LustreError: 218905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6613.112616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6613.511373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6617.930231] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6626.378754] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6632.244225] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6639.106304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6639.621797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6642.721727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6646.811411] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6655.683732] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6664.459697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6664.524235] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6664.679482] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6671.443501] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6679.284907] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6681.222899] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6693.197581] Lustre: Permanently deactivating lustre-OST0001 [ 6693.220979] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6701.600604] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6708.068126] Lustre: Permanently reactivating lustre-OST0001 [ 6708.085396] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6708.091415] Lustre: Skipped 2 previous similar messages [ 6710.303362] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6710.317215] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6710.324837] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6710.332950] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6710.361360] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6720.405036] Lustre: server umount lustre-OST0001 complete [ 6720.480518] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6720.487760] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6720.495799] Lustre: Skipped 1 previous similar message [ 6720.497701] LustreError: 220726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6720.998611] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6724.576314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6724.584324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6724.585919] LustreError: Skipped 1 previous similar message [ 6724.614725] Lustre: Skipped 1 previous similar message [ 6724.616589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6726.117830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6726.119897] LustreError: 220726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6726.122470] Lustre: Skipped 2 previous similar messages [ 6726.157726] LustreError: 220726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6731.247714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6731.253448] Lustre: Skipped 1 previous similar message [ 6731.253695] LustreError: 220728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6731.280319] LustreError: 220728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6736.358964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6736.373344] Lustre: Skipped 1 previous similar message [ 6738.399144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6738.579589] Lustre: server umount lustre-OST0000 complete [ 6746.593602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.597257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6746.610473] Lustre: Skipped 1 previous similar message [ 6746.612687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6748.757093] Lustre: server umount lustre-MDT0000 complete [ 6751.712974] LustreError: 218900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6751.740365] LustreError: 218900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6752.262478] LustreError: 222693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384296 with bad export cookie 882788664116509608 [ 6752.273621] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6752.278800] LustreError: 222693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6752.591728] Lustre: server umount lustre-MDT0001 complete [ 6764.220935] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6766.969814] Key type lgssc unregistered [ 6767.274227] LNet: 223957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6767.285700] LNetError: 223957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6767.299469] LNet: Removed LNI 192.168.203.116@tcp [ 6767.980534] Key type .llcrypt unregistered [ 6767.984899] Key type ._llcrypt unregistered [ 6810.151357] Key type ._llcrypt registered [ 6810.154041] Key type .llcrypt registered [ 6810.259527] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6811.725662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6811.753032] alg: No test for adler32 (adler32-zlib) [ 6813.004851] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 6813.234689] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6814.911175] Key type lgssc registered [ 6816.167071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6825.606206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6825.642887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6826.970166] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6827.020114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6827.196977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6831.538488] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6840.822333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.989482] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6841.056668] Lustre: 226164:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6841.365520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6846.767875] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6854.921074] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6860.900212] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6868.212905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6868.417664] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6868.708159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6875.902766] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6877.374802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6884.211756] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6888.438782] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6891.874253] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6900.937687] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6905.671769] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6913.742171] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6935.789700] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6935.793183] Lustre: Skipped 2 previous similar messages [ 6973.409242] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6973.415123] Lustre: Skipped 4 previous similar messages [ 7037.440594] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7037.497158] Lustre: Skipped 6 previous similar messages [ 7067.105671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7067.126493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7067.134176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7067.151109] Lustre: Skipped 1 previous similar message [ 7072.225106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7072.236345] Lustre: Skipped 1 previous similar message [ 7072.448973] Lustre: server umount lustre-OST0000 complete [ 7076.833181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7076.849079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.866784] Lustre: Skipped 1 previous similar message [ 7076.876232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7082.362048] Lustre: server umount lustre-MDT0000 complete [ 7086.469694] LustreError: 226914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384630 with bad export cookie 2671995887780660056 [ 7086.474283] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7086.482452] LustreError: 226914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7087.005477] Lustre: server umount lustre-MDT0001 complete [ 7098.797751] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7101.574270] Key type lgssc unregistered [ 7101.853926] LNet: 230971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7101.861036] LNetError: 230971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7101.882982] LNet: Removed LNI 192.168.203.116@tcp [ 7102.680232] Key type .llcrypt unregistered [ 7102.684179] Key type ._llcrypt unregistered [ 7128.959847] Key type ._llcrypt registered [ 7128.962095] Key type .llcrypt registered [ 7129.025921] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7129.582668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7129.606063] alg: No test for adler32 (adler32-zlib) [ 7130.668702] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 7130.915797] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7132.591203] Key type lgssc registered [ 7134.083981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7144.381550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7144.416350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7146.303914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7152.323699] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7156.717765] LustreError: 232386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7156.745462] LustreError: 232386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7161.828420] LustreError: 232385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7162.073806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7162.513276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7166.454139] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7175.727741] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7182.452152] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7189.267820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7189.697143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7191.735931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7196.381543] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7204.451880] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7222.756891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7222.761370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7222.764621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7222.776880] LustreError: Skipped 1 previous similar message [ 7222.785183] Lustre: Skipped 1 previous similar message [ 7227.746816] Lustre: server umount lustre-OST0000 complete [ 7233.015945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7233.030937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7233.033070] Lustre: Skipped 1 previous similar message [ 7233.054855] Lustre: Skipped 2 previous similar messages [ 7238.119086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7238.131049] Lustre: Skipped 1 previous similar message [ 7242.210422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7242.217097] Lustre: Skipped 1 previous similar message [ 7247.328157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7247.332447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7247.350327] Lustre: Skipped 1 previous similar message [ 7247.640257] Lustre: server umount lustre-MDT0000 complete [ 7251.854155] LustreError: 232367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384795 with bad export cookie 6240057132768002207 [ 7251.857489] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7251.870706] LustreError: 232367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7252.390980] Lustre: server umount lustre-MDT0001 complete [ 7265.287969] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7268.395366] Key type lgssc unregistered [ 7268.765795] LNet: 236078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7268.773755] LNetError: 236078:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7268.793747] LNet: Removed LNI 192.168.203.116@tcp [ 7269.819516] Key type .llcrypt unregistered [ 7269.824900] Key type ._llcrypt unregistered [ 7294.625812] Key type ._llcrypt registered [ 7294.635598] Key type .llcrypt registered [ 7294.752386] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7296.049360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7296.094449] alg: No test for adler32 (adler32-zlib) [ 7297.256854] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 7297.505543] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7299.191170] Key type lgssc registered [ 7300.509716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7573.820899] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7576.935522] Key type lgssc unregistered [ 7577.256793] LNet: 238103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7577.270322] LNetError: 238103:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7577.306493] LNet: Removed LNI 192.168.203.116@tcp [ 7578.102922] Key type .llcrypt unregistered [ 7578.104959] Key type ._llcrypt unregistered [ 7604.676284] Key type ._llcrypt registered [ 7604.680470] Key type .llcrypt registered [ 7604.811481] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 7621.953254] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7623.021285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7623.105751] alg: No test for adler32 (adler32-zlib) [ 7624.093364] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 7624.308910] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7626.007743] Key type lgssc registered [ 7627.346604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7659.974492] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7391 sec ========== 03:53:23 (1787385203) [ 7661.659241] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:53:25 (1787385205) === [ 7665.851363] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:53:28 (1787385208) === [ 7712.271934] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7715.323176] Key type lgssc unregistered [ 7715.743907] LNet: 243339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7715.759839] LNetError: 243339:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7715.798159] LNet: Removed LNI 192.168.203.116@tcp [ 7716.943367] Key type .llcrypt unregistered [ 7716.949065] Key type ._llcrypt unregistered