[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 528806070 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002314] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.006395] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008037] pid_max: default: 32768 minimum: 301 [ 0.009185] LSM: Security Framework initializing [ 0.010078] Yama: becoming mindful. [ 0.012053] SELinux: Initializing. [ 0.013098] *** VALIDATE selinux *** [ 0.021762] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026480] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027226] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028157] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] *** VALIDATE tmpfs *** [ 0.032073] *** VALIDATE proc *** [ 0.033316] *** VALIDATE cgroup *** [ 0.034016] *** VALIDATE cgroup2 *** [ 0.035370] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036229] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038039] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042097] debug: unmapping init [mem 0xffffffff85459000-0xffffffff85460fff] [ 0.045185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046870] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047035] ... version: 2 [ 0.048021] ... bit width: 48 [ 0.049023] ... generic registers: 4 [ 0.050022] ... value mask: 0000ffffffffffff [ 0.051026] ... max period: 00007fffffffffff [ 0.052039] ... fixed-purpose events: 3 [ 0.053029] ... event mask: 000000070000000f [ 0.054452] rcu: Hierarchical SRCU implementation. [ 0.056694] smp: Bringing up secondary CPUs ... [ 0.057758] x86: Booting SMP configuration: [ 0.058034] .... node #0, CPUs: #1 #2 #3 [ 0.061498] smp: Brought up 1 node, 4 CPUs [ 0.063017] smpboot: Max logical packages: 1 [ 0.064024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136206] node 0 deferred pages initialised in 70ms [ 0.140124] devtmpfs: initialized [ 0.141317] x86/mm: Memory block size: 128MB [ 0.144928] gcov: version magic: 0x41383552 [ 0.148360] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149113] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150360] pinctrl core: initialized pinctrl subsystem [ 0.152278] [ 0.153009] ************************************************************* [ 0.155022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157018] ** ** [ 0.159015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162017] ** ** [ 0.164016] ** This means that this kernel is built to expose internal ** [ 0.167018] ** IOMMU data structures, which may compromise security on ** [ 0.169018] ** your system. ** [ 0.172023] ** ** [ 0.174014] ** If you see this message and you are not debugging the ** [ 0.177017] ** kernel, report this immediately to your vendor! ** [ 0.179014] ** ** [ 0.181021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184018] ************************************************************* [ 0.187484] NET: Registered protocol family 16 [ 0.189675] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199128] cpuidle: using governor menu [ 0.200811] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201752] PCI: Using configuration type 1 for base access [ 0.202166] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209156] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216151] cryptd: max_cpu_qlen set to 1000 [ 0.220249] ACPI: Added _OSI(Module Device) [ 0.221020] ACPI: Added _OSI(Processor Device) [ 0.223020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230607] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.238351] ACPI: Interpreter enabled [ 0.240068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241017] ACPI: Using IOAPIC for interrupt routing [ 0.243149] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247503] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257972] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.260057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.271411] acpiphp: Slot [2] registered [ 0.273290] acpiphp: Slot [5] registered [ 0.275153] acpiphp: Slot [6] registered [ 0.276189] acpiphp: Slot [7] registered [ 0.278191] acpiphp: Slot [8] registered [ 0.279117] acpiphp: Slot [9] registered [ 0.280089] acpiphp: Slot [10] registered [ 0.282107] acpiphp: Slot [3] registered [ 0.283118] acpiphp: Slot [4] registered [ 0.285099] acpiphp: Slot [11] registered [ 0.286076] acpiphp: Slot [12] registered [ 0.287090] acpiphp: Slot [13] registered [ 0.288078] acpiphp: Slot [14] registered [ 0.289065] acpiphp: Slot [15] registered [ 0.290028] acpiphp: Slot [16] registered [ 0.291052] acpiphp: Slot [17] registered [ 0.292070] acpiphp: Slot [18] registered [ 0.293103] acpiphp: Slot [19] registered [ 0.294062] acpiphp: Slot [20] registered [ 0.295045] acpiphp: Slot [21] registered [ 0.295956] acpiphp: Slot [22] registered [ 0.297095] acpiphp: Slot [23] registered [ 0.298074] acpiphp: Slot [24] registered [ 0.299068] acpiphp: Slot [25] registered [ 0.300079] acpiphp: Slot [26] registered [ 0.301047] acpiphp: Slot [27] registered [ 0.301999] acpiphp: Slot [28] registered [ 0.303077] acpiphp: Slot [29] registered [ 0.304122] acpiphp: Slot [30] registered [ 0.305081] acpiphp: Slot [31] registered [ 0.306034] PCI host bridge to bus 0000:00 [ 0.306941] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.314027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.316026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.317221] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.319859] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.322056] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.331021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.335091] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.339029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.348218] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.354968] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.358055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362014] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.369022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.380987] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.386021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.393031] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.400028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.432022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.455264] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.470022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.488025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.502162] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.510020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.517025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.541032] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.549711] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.560028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.569022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.611022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.622000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.635024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.646030] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.664028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.678169] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.688023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.699025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.717031] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.729299] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.733461] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.736532] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.738514] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.741288] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.746111] iommu: Default domain type: Passthrough [ 0.750490] SCSI subsystem initialized [ 0.752164] ACPI: bus type USB registered [ 0.753199] usbcore: registered new interface driver usbfs [ 0.755103] usbcore: registered new interface driver hub [ 0.757129] usbcore: registered new device driver usb [ 0.760250] pps_core: LinuxPPS API ver. 1 registered [ 0.762013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.765099] PTP clock support registered [ 0.768095] EDAC MC: Ver: 3.0.0 [ 0.770209] PCI: Using ACPI for IRQ routing [ 0.772987] NetLabel: Initializing [ 0.775019] NetLabel: domain hash size = 128 [ 0.777013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.779136] NetLabel: unlabeled traffic allowed by default [ 0.782162] vgaarb: loaded [ 0.784327] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.785013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.789257] clocksource: Switched to clocksource kvm-clock [ 0.894397] VFS: Disk quotas dquot_6.6.0 [ 0.896040] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.898413] *** VALIDATE ramfs *** [ 0.899664] *** VALIDATE hugetlbfs *** [ 0.901917] pnp: PnP ACPI init [ 0.904468] pnp: PnP ACPI: found 6 devices [ 0.929177] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.932886] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.935126] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.936851] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.939280] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.941817] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.945037] NET: Registered protocol family 2 [ 0.947967] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.952920] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.955943] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.961670] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.964640] TCP: Hash tables configured (established 65536 bind 65536) [ 0.968152] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.971162] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.974462] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.977102] NET: Registered protocol family 1 [ 0.979762] RPC: Registered named UNIX socket transport module. [ 0.981703] RPC: Registered udp transport module. [ 0.983118] RPC: Registered tcp transport module. [ 0.984535] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.986698] NET: Registered protocol family 44 [ 0.988150] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.989947] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.991889] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.993849] PCI: CLS 0 bytes, default 64 [ 0.996163] Unpacking initramfs... [ 2.394264] debug: unmapping init [mem 0xffff9396fcc54000-0xffff9396fffbffff] [ 2.397795] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.400282] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.402924] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.147489] Initialise system trusted keyrings [ 3.149123] Key type blacklist registered [ 3.153642] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.163156] zbud: loaded [ 3.166119] *** VALIDATE nfs *** [ 3.167470] *** VALIDATE nfs4 *** [ 3.169126] pstore: using deflate compression [ 3.172897] Platform Keyring initialized [ 3.310735] NET: Registered protocol family 38 [ 3.313538] Key type asymmetric registered [ 3.315885] Asymmetric key parser 'x509' registered [ 3.319195] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.323507] io scheduler mq-deadline registered [ 3.325067] io scheduler kyber registered [ 3.327561] io scheduler bfq registered [ 3.333759] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.340048] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.348157] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.352942] ACPI: Power Button [PWRF] [ 3.358617] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.368948] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.390324] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.400591] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.423107] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.454036] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.485434] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.491903] Non-volatile memory driver v1.3 [ 3.494129] Linux agpgart interface v0.103 [ 3.539172] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.543970] vda: detected capacity change from 0 to 76681216 [ 3.565174] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.569333] vdb: detected capacity change from 0 to 1073741824 [ 3.591691] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.596480] vdc: detected capacity change from 0 to 2621440000 [ 3.623195] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.627235] vdd: detected capacity change from 0 to 2621440000 [ 3.652724] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.657175] vde: detected capacity change from 0 to 4294967296 [ 3.677501] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.681600] vdf: detected capacity change from 0 to 4294967296 [ 3.692296] libphy: Fixed MDIO Bus: probed [ 3.702191] usbcore: registered new interface driver usbserial_generic [ 3.708317] usbserial: USB Serial support registered for generic [ 3.712731] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.724072] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.726293] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.729199] mousedev: PS/2 mouse device common for all mice [ 3.734408] rtc_cmos 00:05: RTC can wake from S4 [ 3.738654] rtc_cmos 00:05: registered as rtc0 [ 3.738942] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.743108] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.748412] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.753468] intel_pstate: CPU model not supported [ 3.758184] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.766686] hid: raw HID events driver (C) Jiri Kosina [ 3.768856] usbcore: registered new interface driver usbhid [ 3.772584] usbhid: USB HID core driver [ 3.775932] drop_monitor: Initializing network drop monitor service [ 3.779468] Initializing XFRM netlink socket [ 3.781732] NET: Registered protocol family 10 [ 3.784663] Segment Routing with IPv6 [ 3.785906] NET: Registered protocol family 17 [ 3.787658] mpls_gso: MPLS GSO support [ 3.792816] RAS: Correctable Errors collector initialized. [ 3.794968] AVX version of gcm_enc/dec engaged. [ 3.796753] AES CTR mode by8 optimization enabled [ 3.879069] sched_clock: Marking stable (3879013957, 0)->(4898582402, -1019568445) [ 3.882282] registered taskstats version 1 [ 3.883965] Loading compiled-in X.509 certificates [ 3.886527] zswap: loaded using pool lzo/zbud [ 3.921479] Key type big_key registered [ 3.944731] Key type encrypted registered [ 3.948530] ima: No TPM chip found, activating TPM-bypass! [ 3.952108] ima: Allocated hash algorithm: sha1 [ 3.954335] ima: No architecture policies found [ 3.956964] evm: Initialising EVM extended attributes: [ 3.961152] evm: security.selinux [ 3.962993] evm: security.ima [ 3.965077] evm: security.capability [ 3.967027] evm: HMAC attrs: 0x1 [ 3.974947] rtc_cmos 00:05: setting system clock to 2026-09-05 20:35:23 UTC (1788640523) [ 3.984731] debug: unmapping init [mem 0xffffffff86403000-0xffffffff865fffff] [ 3.990120] debug: unmapping init [mem 0xffffffff85182000-0xffffffff85458fff] [ 4.003122] Write protecting the kernel read-only data: 28672k [ 4.007691] debug: unmapping init [mem 0xffffffff83803000-0xffffffff839fffff] [ 4.012046] debug: unmapping init [mem 0xffffffff84114000-0xffffffff841fffff] [ 4.111730] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.126904] systemd[1]: Detected virtualization kvm. [ 4.130172] systemd[1]: Detected architecture x86-64. [ 4.133210] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.186284] systemd[1]: No hostname configured. [ 4.190769] systemd[1]: Set hostname to . [ 4.196387] random: systemd: uninitialized urandom read (16 bytes read) [ 4.199738] systemd[1]: Initializing machine ID from random generator. [ 4.593156] random: systemd: uninitialized urandom read (16 bytes read) [ 4.596470] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.616797] random: systemd: uninitialized urandom read (16 bytes read) [ 4.628933] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.649466] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.576769] device-mapper: uevent: version 1.0.3 [ 6.591375] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.280046] random: fast init done [ 8.340134] virtio_net virtio0 ens2: renamed from eth0 [ 8.582490] scsi host0: ata_piix [ 8.682499] scsi host1: ata_piix [ 8.818217] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.826636] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.487408] random: crng init done [ 13.494263] random: 7 urandom warning(s) missed due to ratelimiting [ 16.067579] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.497651] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.356072] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.705281] SELinux: Disabled at runtime. [ 23.876390] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.900250] systemd[1]: Detected virtualization kvm. [ 23.906089] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.394597] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.407720] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.415239] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.418984] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.422184] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.431321] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.448893] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ 26.742297] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 28.716835] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.465981] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.681312] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.219027] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.365239] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 36.135630] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 36.853938] NFS: Registering the id_resolver key type [ 36.856563] Key type id_resolver registered [ 36.858262] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg205-server login: [ 110.685409] hrtimer: interrupt took 12544053 ns [ 113.638699] libcfs: loading out-of-tree module taints kernel. [ 113.712337] Key type ._llcrypt registered [ 113.714975] Key type .llcrypt registered [ 113.810889] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 137.847824] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 139.927661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 139.970304] alg: No test for adler32 (adler32-zlib) [ 141.903484] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 143.329411] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 145.143170] Key type lgssc registered [ 147.628724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 168.153453] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 209.581319] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 223.234824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 223.260499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.587671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 224.646438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 224.806784] Lustre: lustre-MDT0000: new disk, initializing [ 224.869231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.883973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 230.113572] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.876789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 247.949152] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 247.979209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 247.984829] Lustre: Skipped 1 previous similar message [ 248.111806] Lustre: lustre-MDT0001: new disk, initializing [ 248.202329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 248.242632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 248.258659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 252.865794] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.965298] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 267.483810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.917720] Lustre: lustre-OST0000: new disk, initializing [ 267.922869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 267.931838] Lustre: 8457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 268.053499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 270.590393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 270.617315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 270.738130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 275.669560] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.968968] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 294.310746] Lustre: lustre-OST0001: new disk, initializing [ 294.330468] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 294.338382] Lustre: 9532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 294.426420] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 300.072823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 300.088670] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 300.157165] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 302.219511] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 314.501695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 324.327278] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 329.844285] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing check_logdir /tmp/testlogs/ [ 335.466183] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing yml_node [ 339.440100] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 342.052718] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 344.162560] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 345.527281] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 16:41:03 EDT 2026 [ 362.698129] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 364.643431] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 381.410496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 381.434554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 381.447646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 386.843769] Lustre: server umount lustre-MDT0000 complete [ 391.111902] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640910 with bad export cookie 12426754230018792516 [ 391.114097] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 391.121260] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 392.160188] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 392.161244] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.161889] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 392.161895] Lustre: Skipped 3 previous similar messages [ 392.186291] Lustre: Skipped 4 previous similar messages [ 392.224507] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 396.200594] LustreError: 6528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.245706] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 396.250745] Lustre: Skipped 2 previous similar messages [ 397.602552] Lustre: server umount lustre-MDT0001 complete [ 402.213248] Lustre: server umount lustre-OST0000 complete [ 407.269383] Lustre: server umount lustre-OST0001 complete [ 415.046387] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 426.851255] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 467.275788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.504803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 467.524066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 467.588431] Lustre: lustre-MDT0000: new disk, initializing [ 467.647376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.673621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 473.913634] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 487.128694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.285298] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 487.335920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 487.339563] Lustre: Skipped 1 previous similar message [ 487.403324] Lustre: lustre-MDT0001: new disk, initializing [ 487.508977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 487.549718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 487.569721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 491.778840] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 501.309780] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 507.791845] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 515.216203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 515.538342] Lustre: lustre-OST0000: new disk, initializing [ 515.541291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 515.555545] Lustre: 17058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 515.720459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.102602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 517.140348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 517.353376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 522.793250] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 533.278786] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 538.970515] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 539.272993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 545.150344] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 545.439110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 548.322797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.330138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 548.330382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 548.341324] Lustre: Skipped 1 previous similar message [ 553.479876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 553.484372] Lustre: Skipped 2 previous similar messages [ 554.585656] Lustre: server umount lustre-OST0000 complete [ 558.563933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.684924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.697406] Lustre: Skipped 3 previous similar messages [ 564.887247] Lustre: server umount lustre-MDT0000 complete [ 568.804739] LustreError: 15099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.829567] LustreError: 15099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 568.888854] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641088 with bad export cookie 12426754230018794049 [ 568.895310] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 568.899528] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 569.343572] Lustre: server umount lustre-MDT0001 complete [ 572.955506] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 16:44:51 (1788641091) [ 579.609423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.325907] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.423108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 584.692798] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 585.708427] LustreError: 19326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.484554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.568276] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 610.362833] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 615.494453] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 621.814777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 628.188874] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 634.990374] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 645.104291] Lustre: server umount lustre-OST0000 complete [ 650.210725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.214041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.227540] Lustre: Skipped 2 previous similar messages [ 659.581071] Lustre: server umount lustre-MDT0000 complete [ 660.449559] LustreError: 20243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 660.473727] LustreError: 20243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 661.471393] Lustre: 3655:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641164/real 1788641164] req@ffff9397412aad80 x1875525670971392/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788641180 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 661.472259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.485919] Lustre: 3655:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 663.954168] LustreError: 19307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641183 with bad export cookie 12426754230018794931 [ 663.963702] LustreError: 19307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 663.965423] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 664.411691] Lustre: server umount lustre-MDT0001 complete [ 676.273189] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 679.613840] Key type lgssc unregistered [ 679.886388] LNet: 22858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.892318] LNetError: 22858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.924250] LNet: Removed LNI 192.168.202.105@tcp [ 681.672343] Key type .llcrypt unregistered [ 681.679974] Key type ._llcrypt unregistered [ 710.729695] Key type ._llcrypt registered [ 710.734755] Key type .llcrypt registered [ 710.854191] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 712.679190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 712.779085] alg: No test for adler32 (adler32-zlib) [ 713.927709] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 714.089477] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 715.775406] Key type lgssc registered [ 716.794612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 725.345921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 725.389412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.093808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 731.842189] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 737.256369] LustreError: 24269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 737.283150] LustreError: 24269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 742.299895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.375520] LustreError: 24268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 742.895046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 747.171666] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 754.770300] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 761.157642] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 768.047269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 768.428145] LustreError: 26092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 768.547784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 771.654657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 775.345810] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 783.467427] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 800.809364] Lustre: server umount lustre-OST0000 complete [ 807.392572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 807.394689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 807.422021] Lustre: Skipped 1 previous similar message [ 807.443376] Lustre: Skipped 1 previous similar message [ 812.512580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 812.526881] Lustre: Skipped 1 previous similar message [ 814.899283] Lustre: server umount lustre-MDT0000 complete [ 817.631403] Lustre: 23580:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641321/real 1788641321] req@0000000019c6e80f x1875526269564672/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788641337 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 817.632819] LustreError: 24269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 817.634340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 817.686341] Lustre: 23580:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 817.769460] LustreError: 24269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 820.238407] LustreError: 27495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641339 with bad export cookie 755425610066351189 [ 820.240233] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 820.292527] LustreError: 27495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 820.709120] Lustre: server umount lustre-MDT0001 complete [ 834.483867] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 837.730811] Key type lgssc unregistered [ 837.993352] LNet: 28248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.000916] LNetError: 28248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.018334] LNet: Removed LNI 192.168.202.105@tcp [ 839.082854] Key type .llcrypt unregistered [ 839.085596] Key type ._llcrypt unregistered [ 869.553518] Key type ._llcrypt registered [ 869.561547] Key type .llcrypt registered [ 869.803089] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 871.552212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 871.767903] alg: No test for adler32 (adler32-zlib) [ 872.854658] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 873.051276] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 874.695203] Key type lgssc registered [ 876.127063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 885.837387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 885.895308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.776442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.205423] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 898.017696] LustreError: 29660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 898.034810] LustreError: 29660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 903.138117] LustreError: 29661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 903.722899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.184881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 909.970133] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 917.748676] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 924.384443] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 937.689958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 938.181848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.025653] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 953.656364] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 957.636146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 965.098803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 965.112462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.144282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 968.175609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 968.175800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 968.180870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.222407] Lustre: Skipped 1 previous similar message [ 970.729781] Lustre: server umount lustre-OST0000 complete [ 975.842180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 975.861500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.883339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 978.411651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 978.413721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.227738] Lustre: server umount lustre-MDT0000 complete [ 983.522493] LustreError: 29660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 983.550657] LustreError: 29660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 985.927088] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641505 with bad export cookie 8110022675703290353 [ 985.935868] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 985.942583] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 986.318776] Lustre: server umount lustre-MDT0001 complete [ 1000.227911] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1003.758140] Key type lgssc unregistered [ 1004.030250] LNet: 33540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.039635] LNetError: 33540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.071590] LNet: Removed LNI 192.168.202.105@tcp [ 1005.090171] Key type .llcrypt unregistered [ 1005.093143] Key type ._llcrypt unregistered [ 1035.245888] Key type ._llcrypt registered [ 1035.247643] Key type .llcrypt registered [ 1035.385460] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1037.095517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1037.129404] alg: No test for adler32 (adler32-zlib) [ 1038.177220] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1038.484278] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1040.223254] Key type lgssc registered [ 1041.529909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1051.431107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1051.472755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1053.079025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.704801] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1063.393644] LustreError: 34949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.416834] LustreError: 34949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1068.515270] LustreError: 34950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1068.708489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.280214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1075.790525] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1084.405048] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1092.045512] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1099.977345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1100.259241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.633651] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1116.765476] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1122.843773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1138.150948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1138.171335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.208489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1138.228847] Lustre: Skipped 1 previous similar message [ 1142.276501] Lustre: server umount lustre-OST0000 complete [ 1148.407073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.427699] Lustre: Skipped 1 previous similar message [ 1148.435646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1148.440723] Lustre: Skipped 1 previous similar message [ 1153.506971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.526173] Lustre: Skipped 1 previous similar message [ 1154.079975] Lustre: server umount lustre-MDT0000 complete [ 1157.608527] LustreError: 34949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.628580] LustreError: 34949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1158.353939] LustreError: 34930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641677 with bad export cookie 6828658770232899352 [ 1158.355171] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1158.362592] LustreError: 34930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1158.912571] Lustre: server umount lustre-MDT0001 complete [ 1171.579849] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1175.443765] Key type lgssc unregistered [ 1175.837537] LNet: 38507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.855683] LNetError: 38507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.881249] LNet: Removed LNI 192.168.202.105@tcp [ 1177.078089] Key type .llcrypt unregistered [ 1177.082365] Key type ._llcrypt unregistered [ 1202.906337] Key type ._llcrypt registered [ 1202.907930] Key type .llcrypt registered [ 1203.093306] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1204.354136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1204.525527] alg: No test for adler32 (adler32-zlib) [ 1205.768895] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1206.142382] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1207.960387] Key type lgssc registered [ 1209.734656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1219.833439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1219.861115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.737865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1227.144758] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1232.354146] LustreError: 39918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.392894] LustreError: 39918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1236.593159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.148058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1242.707099] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1251.590562] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1258.344409] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1266.992666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1267.177751] LustreError: 41741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.403707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1272.843255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1274.803403] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1282.946837] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1289.701502] Lustre: lustre-OST0000: Not available for connect from 192.168.202.5@tcp (stopping) [ 1292.259860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.262223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1292.273577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1294.744712] Lustre: lustre-OST0000: Not available for connect from 192.168.202.5@tcp (stopping) [ 1294.757241] Lustre: Skipped 1 previous similar message [ 1295.214764] Lustre: server umount lustre-OST0000 complete [ 1316.058928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.075395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1316.105632] Lustre: Skipped 2 previous similar messages [ 1321.440461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1321.448920] Lustre: Skipped 2 previous similar messages [ 1322.711731] Lustre: server umount lustre-MDT0000 complete [ 1326.562385] LustreError: 39914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.583073] LustreError: 39914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1326.785981] LustreError: 39898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641846 with bad export cookie 1702844506472689593 [ 1326.801056] LustreError: 39898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1326.802797] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1333.506314] Lustre: server umount lustre-MDT0001 complete [ 1345.925909] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1348.680416] Key type lgssc unregistered [ 1349.026430] LNet: 43626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.038207] LNetError: 43626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.058619] LNet: Removed LNI 192.168.202.105@tcp [ 1350.054520] Key type .llcrypt unregistered [ 1350.057458] Key type ._llcrypt unregistered [ 1377.971983] Key type ._llcrypt registered [ 1377.975422] Key type .llcrypt registered [ 1378.125927] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1379.575621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1379.816167] alg: No test for adler32 (adler32-zlib) [ 1381.036570] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1381.361507] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1383.031199] Key type lgssc registered [ 1384.861781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1396.636353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1396.652548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1398.103527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.344206] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1408.491607] LustreError: 45039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1408.516133] LustreError: 45039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1413.602642] LustreError: 45040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1414.360181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.887492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1420.378565] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1429.593609] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1436.018283] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1444.480838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1444.911881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1451.378204] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1458.999459] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1469.521330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1476.015727] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.5@tcp (stopping) [ 1476.579725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1476.583221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.604913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1477.738101] Lustre: server umount lustre-MDT0000 complete [ 1479.657286] LustreError: 45040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.660141] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.277380] LustreError: 45019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642001 with bad export cookie 5405998569979928550 [ 1482.288977] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1484.789467] LustreError: 46013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.790374] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1484.801906] LustreError: 46013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1484.807863] Lustre: Skipped 1 previous similar message [ 1484.809298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1487.778888] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1487.780957] Lustre: Skipped 1 previous similar message [ 1488.720062] Lustre: server umount lustre-MDT0001 complete [ 1519.072117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1519.351319] Lustre: server umount lustre-OST0000 complete [ 1538.771228] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1542.557841] Key type lgssc unregistered [ 1543.014921] LNet: 48937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.028050] LNetError: 48937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.059136] LNet: Removed LNI 192.168.202.105@tcp [ 1544.136128] Key type .llcrypt unregistered [ 1544.139243] Key type ._llcrypt unregistered [ 1573.474614] Key type ._llcrypt registered [ 1573.476877] Key type .llcrypt registered [ 1573.672735] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1575.455907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1575.491288] alg: No test for adler32 (adler32-zlib) [ 1576.620458] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1577.001242] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1578.655217] Key type lgssc registered [ 1580.096675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1587.985262] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1588.010602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1603.551490] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1608.671619] LustreError: 50319:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 1635.361088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.432416] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1650.175754] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1815.924506] Lustre: server umount lustre-OST0000 complete [ 1834.194376] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1836.827735] Key type lgssc unregistered [ 1837.181538] LNet: 51993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.193298] LNetError: 51993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.214187] LNet: Removed LNI 192.168.202.105@tcp [ 1838.237364] Key type .llcrypt unregistered [ 1838.240831] Key type ._llcrypt unregistered [ 1864.118336] Key type ._llcrypt registered [ 1864.124658] Key type .llcrypt registered [ 1864.280184] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1865.989417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1866.222761] alg: No test for adler32 (adler32-zlib) [ 1867.514515] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1868.017295] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1869.787614] Key type lgssc registered [ 1871.791325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1883.329250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1883.351993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.947616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.738436] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1895.396380] LustreError: 53408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.436019] LustreError: 53408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1899.707493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.037114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1904.966379] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1912.950369] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1919.785658] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1927.330762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1927.607630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.658299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1933.479991] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1940.475809] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1944.619565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1944.620863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.784775] Lustre: server umount lustre-OST0000 complete [ 1954.787530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.791534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.804686] Lustre: Skipped 1 previous similar message [ 1954.830116] Lustre: Skipped 2 previous similar messages [ 1958.466793] Lustre: server umount lustre-MDT0000 complete [ 1959.907118] LustreError: 54135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1961.841365] LustreError: 53387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642481 with bad export cookie 9506485478594534447 [ 1961.855241] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1961.857881] LustreError: 53387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1962.144750] Lustre: server umount lustre-MDT0001 complete [ 1972.172509] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1974.758582] Key type lgssc unregistered [ 1975.147963] LNet: 56935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.155239] LNetError: 56935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.171545] LNet: Removed LNI 192.168.202.105@tcp [ 1975.948067] Key type .llcrypt unregistered [ 1975.950127] Key type ._llcrypt unregistered [ 1997.838954] Key type ._llcrypt registered [ 1997.841045] Key type .llcrypt registered [ 1997.962144] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1999.278726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1999.303534] alg: No test for adler32 (adler32-zlib) [ 2000.388100] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2000.696038] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2002.378108] Key type lgssc registered [ 2003.556424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2010.790899] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2010.808582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2026.336106] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2031.455347] LustreError: 58309:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 2058.143223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.996938] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2073.356948] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2080.905902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.465935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.545520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2085.908300] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2086.882673] LustreError: 58335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.909838] LustreError: 58335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2092.003594] LustreError: 58334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.034631] LustreError: 59347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.046376] LustreError: 59347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2094.890516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.239642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2099.733307] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2106.774755] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2111.621588] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2115.553688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2115.556885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.564100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.069288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.078392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.964304] Lustre: server umount lustre-OST0000 complete [ 2145.760670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.767358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2145.789482] Lustre: Skipped 1 previous similar message [ 2145.807762] Lustre: Skipped 2 previous similar messages [ 2148.707966] Lustre: server umount lustre-MDT0000 complete [ 2150.889980] LustreError: 59351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.907045] LustreError: 59351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2152.552341] LustreError: 58317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642672 with bad export cookie 2477309767056221887 [ 2152.564163] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2159.105802] Lustre: server umount lustre-MDT0001 complete [ 2173.056767] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2175.918582] Key type lgssc unregistered [ 2176.231542] LNet: 62056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.255772] LNetError: 62056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2176.273523] LNet: Removed LNI 192.168.202.105@tcp [ 2177.134199] Key type .llcrypt unregistered [ 2177.139061] Key type ._llcrypt unregistered [ 2201.545219] Key type ._llcrypt registered [ 2201.546667] Key type .llcrypt registered [ 2201.620935] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2202.840138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2202.933855] alg: No test for adler32 (adler32-zlib) [ 2203.998689] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2204.278880] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2205.999242] Key type lgssc registered [ 2207.223326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2214.549584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2214.578700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2215.998041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2220.512782] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2226.148980] LustreError: 63468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.170988] LustreError: 63468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2228.583387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.956344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2234.118409] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2241.426090] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2247.219318] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2253.826056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2254.472410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.659381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2261.523910] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2269.091613] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2286.562840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2286.583090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.602329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2289.647967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2289.652742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2289.652765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.674613] Lustre: Skipped 1 previous similar message [ 2294.752683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.881552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.895540] Lustre: Skipped 2 previous similar messages [ 2300.383232] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2300.851291] Lustre: server umount lustre-OST0000 complete [ 2304.995244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.017649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2305.025275] Lustre: Skipped 1 previous similar message [ 2311.047254] Lustre: server umount lustre-MDT0000 complete [ 2315.096671] LustreError: 63449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642834 with bad export cookie 178515106905333883 [ 2315.109172] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.110042] LustreError: 63449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2315.548569] Lustre: server umount lustre-MDT0001 complete [ 2327.459767] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2330.123827] Key type lgssc unregistered [ 2330.427625] LNet: 67024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2330.441019] LNetError: 67024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2330.460102] LNet: Removed LNI 192.168.202.105@tcp [ 2331.257200] Key type .llcrypt unregistered [ 2331.263772] Key type ._llcrypt unregistered [ 2358.971964] Key type ._llcrypt registered [ 2358.973818] Key type .llcrypt registered [ 2359.052780] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2360.190607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2360.362987] alg: No test for adler32 (adler32-zlib) [ 2361.530229] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2361.851494] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2363.567147] Key type lgssc registered [ 2364.583488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2373.227613] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 17:14:51 (1788642891) [ 2380.145227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2380.174279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.684313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.835551] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2392.034388] LustreError: 68725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.052100] LustreError: 68725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2396.293082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.692524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2401.618635] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2409.148573] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2416.047291] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2424.478630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2424.875281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2426.929348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2431.176290] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2438.809152] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2448.448149] Lustre: Failing over lustre-MDT0000 [ 2448.862895] Lustre: server umount lustre-MDT0000 complete [ 2451.415115] LustreError: 69713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.471710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.474596] LustreError: 68721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.489268] Lustre: Skipped 1 previous similar message [ 2452.522598] LustreError: 68721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2456.473600] LustreError: 68720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.839987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.106753] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2459.312943] Lustre: *** cfs_fail_loc=135, val=0*** [ 2459.455508] LustreError: 71812:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2459.463536] LustreError: 71812:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.105@tcp: cfg command failed: rc = -2 [ 2459.474647] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2459.474647] [ 2459.484396] LustreError: MGC192.168.202.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2459.504867] LustreError: 71801:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2459.513407] LustreError: 71801:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2459.528569] LustreError: 71801:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2459.536163] LustreError: 71801:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2459.574484] Lustre: server umount lustre-MDT0000 complete [ 2459.578384] LustreError: 71801:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2461.605231] LustreError: 68722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.645299] LustreError: 68722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2466.228643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.373624] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.664701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.759786] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2471.218642] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2471.833613] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2471.912278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2471.992116] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2472.092614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2482.150673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.152924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2482.157670] Lustre: Skipped 2 previous similar messages [ 2486.839930] Lustre: server umount lustre-OST0000 complete [ 2492.384679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2492.390532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.401237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2492.418953] Lustre: Skipped 1 previous similar message [ 2492.419568] Lustre: Skipped 1 previous similar message [ 2497.040274] Lustre: server umount lustre-MDT0000 complete [ 2497.509844] LustreError: 68720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.528439] LustreError: 68720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2500.304971] LustreError: 72639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643019 with bad export cookie 5236845840593905879 [ 2500.314434] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2500.330518] LustreError: 72639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2500.681179] Lustre: server umount lustre-MDT0001 complete [ 2512.585061] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2516.048746] Key type lgssc unregistered [ 2516.451479] LNet: 73593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2516.461321] LNetError: 73593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2516.489248] LNet: Removed LNI 192.168.202.105@tcp [ 2517.562566] Key type .llcrypt unregistered [ 2517.572749] Key type ._llcrypt unregistered [ 2543.434961] Key type ._llcrypt registered [ 2543.441216] Key type .llcrypt registered [ 2543.566578] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2544.955195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2544.975714] alg: No test for adler32 (adler32-zlib) [ 2546.060487] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2546.260817] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2547.911216] Key type lgssc registered [ 2549.126413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2557.553905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2557.577952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.050360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.451836] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2569.201300] LustreError: 75002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.212914] LustreError: 75002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2571.774477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.198229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2576.703490] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2584.121716] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2590.037779] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2596.616746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2597.029463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2603.265551] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2604.223435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2610.638551] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2615.692861] Lustre: Failing over lustre-MDT0000 [ 2615.910343] Lustre: server umount lustre-MDT0000 complete [ 2617.263723] LustreError: 76242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.338829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2618.349591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.365032] LustreError: 75002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.875302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.890352] Lustre: Skipped 1 previous similar message [ 2622.363192] LustreError: 76242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.393664] LustreError: 76242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2625.090645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.226702] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2625.321333] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2625.425947] LustreError: 78086:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2625.435472] LustreError: 78086:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.105@tcp: cfg command failed: rc = -9 [ 2625.444531] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2625.444531] [ 2625.456677] LustreError: MGC192.168.202.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2625.479838] LustreError: 78075:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2625.491544] LustreError: 78075:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2625.501503] LustreError: 78075:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2625.508312] LustreError: 78075:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2625.534244] Lustre: server umount lustre-MDT0000 complete [ 2625.538151] LustreError: 78075:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2627.491888] LustreError: 74998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.518141] LustreError: 74998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2631.526415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.628168] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2631.934868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2631.999658] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2632.616233] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2636.500091] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2637.292226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2637.342547] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2637.438849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2644.448436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2644.471314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2644.505275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2647.530901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2647.546617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.473183] Lustre: server umount lustre-OST0000 complete [ 2657.764963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.765817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2657.793480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.793492] Lustre: Skipped 1 previous similar message [ 2661.911697] Lustre: server umount lustre-MDT0000 complete [ 2662.901516] LustreError: 75002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.931717] LustreError: 75002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2666.433151] LustreError: 74984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643185 with bad export cookie 9950224482825024942 [ 2666.452199] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.458142] LustreError: 74984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2666.870756] Lustre: server umount lustre-MDT0001 complete [ 2678.738659] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2681.870981] Key type lgssc unregistered [ 2682.123645] LNet: 79867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2682.140773] LNetError: 79867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2682.153968] LNet: Removed LNI 192.168.202.105@tcp [ 2683.301331] Key type .llcrypt unregistered [ 2683.306467] Key type ._llcrypt unregistered [ 2697.878366] Key type ._llcrypt registered [ 2697.882840] Key type .llcrypt registered [ 2698.027801] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2698.300637] Key type .llcrypt unregistered [ 2698.306493] Key type ._llcrypt unregistered [ 2721.659537] Key type ._llcrypt registered [ 2721.661516] Key type .llcrypt registered [ 2721.775145] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2723.296640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2723.377171] alg: No test for adler32 (adler32-zlib) [ 2724.454659] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2724.720810] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2726.439170] Key type lgssc registered [ 2727.651698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2736.638296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2736.649495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.194556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.533674] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2748.386579] LustreError: 81763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.418891] LustreError: 81763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2753.449972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.505461] LustreError: 81764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.800527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2758.170813] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2766.216992] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2772.080820] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2780.390629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2780.784499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.854246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2787.036189] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2794.886271] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2809.828793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2809.855082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2811.143873] Lustre: server umount lustre-OST0000 complete [ 2814.945606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.961472] Lustre: Skipped 1 previous similar message [ 2814.968567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2814.983954] Lustre: Skipped 2 previous similar messages [ 2820.066301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2820.078066] Lustre: Skipped 1 previous similar message [ 2825.184462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2825.193134] Lustre: Skipped 1 previous similar message [ 2829.285933] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2829.466452] Lustre: server umount lustre-MDT0000 complete [ 2830.316786] LustreError: 81763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.341312] LustreError: 81763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2832.998406] LustreError: 81745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643352 with bad export cookie 4415716190781138403 [ 2833.030898] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2833.490686] Lustre: server umount lustre-MDT0001 complete [ 2844.816584] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2847.426968] Key type lgssc unregistered [ 2847.699728] LNet: 85316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2847.703909] LNetError: 85316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2848.756154] LNet: Removed LNI 192.168.202.105@tcp [ 2849.470154] Key type .llcrypt unregistered [ 2849.472243] Key type ._llcrypt unregistered [ 2870.105863] Key type ._llcrypt registered [ 2870.107347] Key type .llcrypt registered [ 2870.154686] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2871.236573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2871.263059] alg: No test for adler32 (adler32-zlib) [ 2872.379898] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2872.664482] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2874.399174] Key type lgssc registered [ 2875.474428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2884.562584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2884.575133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.115125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.501349] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2896.358691] LustreError: 86705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2896.373717] LustreError: 86705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2900.552070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2900.848940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2904.828651] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2911.152754] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2915.907624] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2922.529628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2922.966763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.079323] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2931.182810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2935.783648] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2944.819515] Lustre: server umount lustre-OST0000 complete [ 2951.649312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.658652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2951.669238] Lustre: Skipped 1 previous similar message [ 2951.692312] Lustre: Skipped 1 previous similar message [ 2956.771360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2956.777962] Lustre: Skipped 1 previous similar message [ 2957.856551] Lustre: server umount lustre-MDT0000 complete [ 2961.150732] LustreError: 86686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643480 with bad export cookie 14372342946788068910 [ 2961.165086] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2970.805807] Lustre: server umount lustre-MDT0001 complete [ 2980.857458] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2983.471311] Key type lgssc unregistered [ 2983.808731] LNet: 90236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2983.826062] LNetError: 90236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2983.862837] LNet: Removed LNI 192.168.202.105@tcp [ 2984.964310] Key type .llcrypt unregistered [ 2984.967876] Key type ._llcrypt unregistered [ 3010.277744] Key type ._llcrypt registered [ 3010.281547] Key type .llcrypt registered [ 3010.411492] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3011.663551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3011.754675] alg: No test for adler32 (adler32-zlib) [ 3012.775245] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 3013.111493] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3014.897834] Key type lgssc registered [ 3015.935237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3024.745745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3024.779317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.410828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.473429] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3035.617515] LustreError: 91647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.645523] LustreError: 91647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3039.353325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.716479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3044.376922] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3052.508350] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3058.454399] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3068.104884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3068.509093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.582574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3075.572128] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3082.513469] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3095.524356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.525546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3095.549527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3095.577289] Lustre: Skipped 1 previous similar message [ 3100.640727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.650061] Lustre: Skipped 1 previous similar message [ 3101.540952] Lustre: server umount lustre-OST0000 complete [ 3105.761675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.775887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3105.776397] Lustre: Skipped 2 previous similar messages [ 3105.786293] Lustre: Skipped 1 previous similar message [ 3110.880839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3110.891160] Lustre: Skipped 1 previous similar message [ 3112.127488] Lustre: server umount lustre-MDT0000 complete [ 3116.005313] LustreError: 92648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.017966] LustreError: 92648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3116.109178] LustreError: 91628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643635 with bad export cookie 16037214237021172887 [ 3116.113451] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3116.116593] LustreError: 91628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3116.761122] Lustre: server umount lustre-MDT0001 complete [ 3128.035632] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3130.619150] Key type lgssc unregistered [ 3130.965533] LNet: 95202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3130.968754] LNetError: 95202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3130.996996] LNet: Removed LNI 192.168.202.105@tcp [ 3131.816261] Key type .llcrypt unregistered [ 3131.824644] Key type ._llcrypt unregistered [ 3156.449724] Key type ._llcrypt registered [ 3156.451931] Key type .llcrypt registered [ 3156.575828] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3157.745414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3157.877053] alg: No test for adler32 (adler32-zlib) [ 3159.037950] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 3159.454259] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3161.151836] Key type lgssc registered [ 3162.545335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3170.392251] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3170.417206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3186.015356] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3191.135338] LustreError: 96585:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 3217.951815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.936567] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3232.399807] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3238.057358] Lustre: server umount lustre-OST0000 complete [ 3247.732186] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 17:29:26 (1788643766) [ 3254.995954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3255.541663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3260.369345] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3260.897655] LustreError: 98227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.924377] LustreError: 98227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3266.024132] LustreError: 98226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.019112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.468115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3273.560911] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3281.087339] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3287.089796] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3293.236635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3293.582490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3300.414734] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3300.785041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3307.625113] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3320.288388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3320.300060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3320.314711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3321.312444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3321.323490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.325482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3321.344704] Lustre: Skipped 1 previous similar message [ 3324.157568] Lustre: server umount lustre-OST0000 complete [ 3331.041059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3331.051450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.087207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3334.415621] Lustre: server umount lustre-MDT0000 complete [ 3336.676788] LustreError: 98222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.710408] LustreError: 98222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3337.544767] LustreError: 98207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643857 with bad export cookie 13578142506882429774 [ 3337.554412] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3338.002517] Lustre: server umount lustre-MDT0001 complete [ 3350.129715] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3352.350246] Key type lgssc unregistered [ 3352.565676] LNet: 101915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3352.571313] LNetError: 101915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3353.640304] LNet: Removed LNI 192.168.202.105@tcp [ 3354.480787] Key type .llcrypt unregistered [ 3354.483836] Key type ._llcrypt unregistered [ 3392.220390] Key type ._llcrypt registered [ 3392.228381] Key type .llcrypt registered [ 3392.352766] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3393.489145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3393.540135] alg: No test for adler32 (adler32-zlib) [ 3394.645530] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 3394.912918] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3396.563130] Key type lgssc registered [ 3397.757236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3406.234055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3406.283350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.806360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.663292] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3418.083125] LustreError: 104096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.108730] LustreError: 104096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3423.191941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.201116] LustreError: 104095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3423.752508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3428.320945] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3436.103968] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3442.465436] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3449.508202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3449.935369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.060136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3456.366604] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3464.558044] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3477.990957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3477.998835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3478.039375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3480.042050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3480.045623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3480.045910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.052032] Lustre: Skipped 1 previous similar message [ 3485.154865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3485.164732] Lustre: Skipped 1 previous similar message [ 3490.271255] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3490.274571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3490.292961] Lustre: Skipped 1 previous similar message [ 3490.510186] Lustre: server umount lustre-OST0000 complete [ 3495.395415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.396409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3495.433389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3495.467664] Lustre: Skipped 1 previous similar message [ 3505.633494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3505.636846] Lustre: Skipped 3 previous similar messages [ 3509.217771] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3509.522609] Lustre: server umount lustre-MDT0000 complete [ 3510.766829] LustreError: 104092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3510.780858] LustreError: 104092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3513.782923] LustreError: 104076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644033 with bad export cookie 5499904538973934845 [ 3513.801309] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3513.803039] LustreError: 104076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3514.272810] Lustre: server umount lustre-MDT0001 complete [ 3525.377963] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3528.117372] Key type lgssc unregistered [ 3528.389871] LNet: 107631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3528.398683] LNetError: 107631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3528.414099] LNet: Removed LNI 192.168.202.105@tcp [ 3529.377325] Key type .llcrypt unregistered [ 3529.383524] Key type ._llcrypt unregistered [ 3549.347407] Key type ._llcrypt registered [ 3549.351342] Key type .llcrypt registered [ 3549.455792] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3550.776322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3550.838759] alg: No test for adler32 (adler32-zlib) [ 3552.081879] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 3552.380901] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3554.111199] Key type lgssc registered [ 3555.195823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3562.527289] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3562.549831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3578.079405] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3583.202116] LustreError: 108874:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 3609.951908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.049528] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3628.646179] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3636.313287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.530365] LustreError: MGC192.168.202.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3636.570161] LustreError: 109896:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3636.582700] LustreError: 109896:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3636.594490] LustreError: 109896:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3636.766344] Lustre: server umount lustre-MDT0000 complete [ 3636.773833] LustreError: 109896:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3655.135829] Lustre: 108211:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788644158/real 1788644158] req@000000007d6ccd6f x1875529245396352/t0(0) o400->MGC192.168.202.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788644174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3655.797799] Lustre: server umount lustre-OST0000 complete [ 3665.549738] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 3675.107743] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3713.877209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.083839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3714.140314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3714.211086] Lustre: lustre-MDT0000: new disk, initializing [ 3714.431270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.447265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3718.700258] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3730.482568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3730.617932] Lustre: 112686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3730.675878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3730.680208] Lustre: Skipped 1 previous similar message [ 3730.755655] Lustre: lustre-MDT0001: new disk, initializing [ 3730.860509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3730.908807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3730.920473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3736.024542] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3746.009631] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3751.897476] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3759.095485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3759.428465] Lustre: lustre-OST0000: new disk, initializing [ 3759.431839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3759.439863] Lustre: 114654:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3759.512136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3765.350368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3765.374036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3765.525844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3766.178906] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3777.904516] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3783.601709] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3783.957479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3789.940972] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3790.207893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3796.449356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3796.454498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.469560] LustreError: Skipped 1 previous similar message [ 3796.473068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3796.494480] Lustre: Skipped 1 previous similar message [ 3799.328483] Lustre: server umount lustre-OST0000 complete [ 3806.695327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3806.696659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3806.729740] Lustre: Skipped 1 previous similar message [ 3806.767310] Lustre: Skipped 2 previous similar messages [ 3809.338919] Lustre: server umount lustre-MDT0000 complete [ 3813.447619] LustreError: 112678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644332 with bad export cookie 6173973974160135900 [ 3813.455206] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3813.460047] LustreError: 112678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3813.786854] Lustre: server umount lustre-MDT0001 complete [ 3823.470436] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 17:39:01 (1788644341) [ 3826.994971] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3845.419765] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 3858.455437] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3899.326459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.823288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3899.891530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3899.995550] Lustre: lustre-MDT0000: new disk, initializing [ 3900.088425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.129445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3905.857546] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3917.942662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.038694] Lustre: 119559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3918.118984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3918.126247] Lustre: Skipped 1 previous similar message [ 3918.183757] Lustre: lustre-MDT0001: new disk, initializing [ 3918.288809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3918.331126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3918.350219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3924.195720] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3933.844356] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3939.753255] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3946.991526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3947.391448] Lustre: lustre-OST0000: new disk, initializing [ 3947.397904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3947.403748] Lustre: 121528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3947.531703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3953.259173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3953.274719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3953.416880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3954.527864] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3964.008769] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3969.330988] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3969.628220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3974.549438] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3974.791306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3979.232457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3979.233463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3979.240353] LustreError: Skipped 1 previous similar message [ 3979.244630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3979.252526] Lustre: Skipped 1 previous similar message [ 3979.271130] Lustre: Skipped 1 previous similar message [ 3984.377389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.391772] Lustre: Skipped 1 previous similar message [ 3989.477190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3989.490088] Lustre: Skipped 1 previous similar message [ 3991.007189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3991.208621] Lustre: server umount lustre-OST0000 complete [ 3995.113447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3995.123132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.715559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3999.719926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.733046] Lustre: Skipped 1 previous similar message [ 4001.271476] Lustre: server umount lustre-MDT0000 complete [ 4004.835647] LustreError: 121052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.857191] LustreError: 121052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4005.074265] LustreError: 119549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644524 with bad export cookie 6173973974160136782 [ 4005.076386] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4005.093911] LustreError: 119549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4005.569742] Lustre: server umount lustre-MDT0001 complete [ 4014.313887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4014.920147] LustreError: 123745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4015.022385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4020.094235] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4020.195183] LustreError: 123746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4020.212198] LustreError: 123746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4025.313446] LustreError: 123746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.769787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.327972] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4043.516576] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4048.686392] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4054.699683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4060.723793] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4067.776652] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4073.927709] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4080.610376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4080.621820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.634378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4080.639900] Lustre: Skipped 1 previous similar message [ 4084.918491] Lustre: server umount lustre-OST0000 complete [ 4095.710156] Lustre: server umount lustre-MDT0000 complete [ 4095.975562] LustreError: 123745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4096.002647] LustreError: 123745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4100.168884] LustreError: 126428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644619 with bad export cookie 6173973974160137664 [ 4100.196676] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4100.200917] LustreError: 126428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4100.681366] Lustre: server umount lustre-MDT0001 complete [ 4112.168296] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 4116.033322] Key type lgssc unregistered [ 4116.494347] LNet: 127383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4116.501645] LNetError: 127383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4116.526492] LNet: Removed LNI 192.168.202.105@tcp [ 4117.400783] Key type .llcrypt unregistered [ 4117.402652] Key type ._llcrypt unregistered [ 4136.080868] Key type ._llcrypt registered [ 4136.085583] Key type .llcrypt registered [ 4136.197418] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 4150.723265] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 4152.032434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4152.082760] alg: No test for adler32 (adler32-zlib) [ 4153.078488] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 4153.279626] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 4154.951252] Key type lgssc registered [ 4156.228489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4193.840491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4193.878056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4195.190600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4195.230755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4195.370175] Lustre: lustre-MDT0000: new disk, initializing [ 4195.475180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4195.505615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4200.015179] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4210.223046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.367633] Lustre: 130537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4210.427238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4210.436417] Lustre: Skipped 1 previous similar message [ 4210.529572] Lustre: lustre-MDT0001: new disk, initializing [ 4210.713435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4210.777701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4210.795785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4215.302345] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4224.540809] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4229.529364] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4236.408593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4236.697831] Lustre: lustre-OST0000: new disk, initializing [ 4236.701208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4236.706241] Lustre: 132505:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4236.772781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4243.319238] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4245.589984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4245.599464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4245.699756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4252.922979] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4258.334506] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4258.703900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4263.849895] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4264.268969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4266.473136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4266.473413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.476597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4266.505185] Lustre: Skipped 1 previous similar message [ 4271.600338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4271.605059] Lustre: Skipped 1 previous similar message [ 4272.841476] Lustre: server umount lustre-OST0000 complete [ 4281.828646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.837611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4281.845882] Lustre: Skipped 1 previous similar message [ 4281.861031] Lustre: Skipped 2 previous similar messages [ 4283.825455] Lustre: server umount lustre-MDT0000 complete [ 4287.502297] LustreError: 130531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644807 with bad export cookie 8720891575285212658 [ 4287.505833] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4287.514385] LustreError: 130531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4287.951559] Lustre: server umount lustre-MDT0001 complete [ 4298.729035] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 17:46:56 (1788644816) [ 4305.389616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4305.874339] LustreError: 134918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4305.899621] LustreError: 134918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4305.966288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4310.841935] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4311.020561] LustreError: 134919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4316.132075] LustreError: 134918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.834053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4320.110685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4325.044118] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4332.459948] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4338.591823] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4340.557746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4340.575708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4344.684767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4344.694155] Lustre: Skipped 2 previous similar messages [ 4346.964635] Lustre: server umount lustre-MDT0000 complete [ 4349.921606] LustreError: 135646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4349.945898] LustreError: 135646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4350.740701] LustreError: 134900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644870 with bad export cookie 8720891575285213533 [ 4350.747826] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4350.751238] LustreError: 134900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4351.235132] Lustre: server umount lustre-MDT0001 complete [ 4359.235219] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 17:47:57 (1788644877) [ 4366.488988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4382.112916] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4387.231470] LustreError: 137333:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 4412.960360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4419.521839] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4426.802251] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4429.320502] Lustre: server umount lustre-OST0000 complete [ 4438.727851] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 17:49:16 (1788644956) [ 4445.670950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.039405] LustreError: 138778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4446.131816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4451.553221] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4456.422979] LustreError: 138779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4456.447273] LustreError: 138779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4461.387753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4461.541134] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4465.837739] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4472.720690] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4479.514587] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4487.746654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.234648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.237675] Lustre: Skipped 1 previous similar message [ 4496.116360] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4504.516410] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4513.256265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4513.265857] LustreError: Skipped 1 previous similar message [ 4513.282572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4513.294408] Lustre: Skipped 1 previous similar message [ 4513.302342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4519.467588] Lustre: server umount lustre-MDT0000 complete [ 4521.968906] LustreError: 138779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4521.988400] LustreError: 138779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4522.828501] LustreError: 138758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645042 with bad export cookie 8720891575285214205 [ 4522.853914] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4522.860970] LustreError: 138758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4523.230370] Lustre: server umount lustre-MDT0001 complete [ 4535.889611] Lustre: server umount lustre-OST0000 complete [ 4543.686973] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 17:51:01 (1788645061) [ 4550.321126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4555.677746] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4556.256875] LustreError: 142433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4556.279120] LustreError: 142433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4564.991620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4565.410183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4565.421394] Lustre: Skipped 1 previous similar message [ 4570.094825] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4578.872818] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4584.741797] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4593.652812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4596.295117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4600.928078] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4608.095605] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4613.133289] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4613.405451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4618.643231] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4618.862458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4622.308304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4622.319153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.327041] Lustre: Skipped 2 previous similar messages [ 4622.330672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4622.334108] Lustre: Skipped 3 previous similar messages [ 4624.866660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4635.105952] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4635.431758] Lustre: server umount lustre-OST0000 complete [ 4640.225838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4640.244664] Lustre: Skipped 2 previous similar messages [ 4645.451685] Lustre: server umount lustre-MDT0000 complete [ 4648.693748] LustreError: 142416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645168 with bad export cookie 8720891575285215269 [ 4648.707726] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4648.711463] LustreError: 142416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4649.000167] Lustre: server umount lustre-MDT0001 complete [ 4656.023769] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 17:52:54 (1788645174) [ 4662.481479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4678.048070] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4683.167699] LustreError: 146589:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 4708.900242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4708.911870] Lustre: Skipped 1 previous similar message [ 4714.953477] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4721.857186] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4727.279444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.622492] LustreError: 146614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4727.639183] LustreError: 146614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4727.819595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4731.675778] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4739.229025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4745.422446] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4754.261326] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4760.729574] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4767.028350] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4767.361075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4773.801759] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4774.120765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4779.500809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4779.503111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4779.514463] Lustre: Skipped 2 previous similar messages [ 4779.522880] Lustre: Skipped 12 previous similar messages [ 4782.483241] Lustre: server umount lustre-MDT0000 complete [ 4786.635847] LustreError: 146596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645306 with bad export cookie 8720891575285216298 [ 4786.645726] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4786.651950] LustreError: 146596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4786.971716] Lustre: server umount lustre-MDT0001 complete [ 4801.494985] Lustre: server umount lustre-OST0000 complete [ 4811.710722] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 17:55:29 (1788645329) [ 4821.275078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4836.898516] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4842.015483] LustreError: 150785:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 4874.135971] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4880.141108] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4885.801353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.151627] LustreError: 150810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4886.172391] LustreError: 150810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4886.318414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4890.270929] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4898.787942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4903.098910] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4910.390629] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4916.588628] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4923.855116] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4924.111896] Lustre: lustre-OST0001: new disk, initializing [ 4924.117790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4924.129827] Lustre: 153658:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4931.193766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4931.197484] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4931.315928] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4932.732659] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4945.613766] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4949.639550] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4949.803914] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4953.613580] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4953.785782] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4957.171833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4957.186611] Lustre: Skipped 1 previous similar message [ 4957.208814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4961.757215] Lustre: server umount lustre-OST0000 complete [ 4967.406910] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4972.849890] Lustre: server umount lustre-OST0001 complete [ 4981.227872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4985.024213] Lustre: server umount lustre-MDT0000 complete [ 4989.899208] LustreError: 150793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645509 with bad export cookie 8720891575285217250 [ 4989.903185] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4989.913814] LustreError: 150793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4990.530973] Lustre: server umount lustre-MDT0001 complete [ 5020.624664] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 17:58:58 (1788645538) [ 5022.104370] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5023.573549] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5025.002357] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 17:59:03 (1788645543) [ 5030.958582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.239496] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5031.266737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5031.443971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5031.452535] Lustre: Skipped 6 previous similar messages [ 5036.338385] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5046.521825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.635822] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5046.692516] Lustre: 157216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5054.099917] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5064.591232] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5070.953879] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5077.837777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5077.975586] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5084.657701] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5086.882072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5093.022613] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5098.038897] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5098.265529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5103.372660] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5103.604110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5107.168065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5107.186432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5107.212286] Lustre: Skipped 4 previous similar messages [ 5111.887798] Lustre: server umount lustre-OST0000 complete [ 5135.929460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5143.881675] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5147.756894] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.764144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.770137] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5147.782728] Lustre: Skipped 1 previous similar message [ 5150.932907] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5155.220387] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5155.443536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5159.498586] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5159.756487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5164.996579] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff893e04b7a800.ost_server_uuid 50 [ 5166.452373] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff893e04b7a800.ost_server_uuid in FULL state after 0 sec [ 5166.638410] LustreError: 159089:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.5@tcp) returned error from blocking AST (req@0000000057c056de x1875529875856896 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000021002b8a/0x7906d4aeb592813c lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.5@tcp remote: 0x75f602bb21e6b585 expref: 5 pid: 157224 timeout: 5266 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5166.657025] LustreError: lustre-MDT0000: A client on nid 192.168.202.5@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5166.663391] LustreError: 157213:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.5@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000021002b8a/0x7906d4aeb592813c lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.5@tcp remote: 0x75f602bb21e6b585 expref: 6 pid: 157224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5172.202259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5172.210849] LustreError: Skipped 1 previous similar message [ 5176.327834] Lustre: server umount lustre-OST0000 complete [ 5194.207165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5194.475408] Lustre: server umount lustre-MDT0000 complete [ 5198.831589] LustreError: 157960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645718 with bad export cookie 8720891575285218433 [ 5198.842021] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5198.847489] LustreError: 157960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5213.152040] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5213.429746] Lustre: server umount lustre-MDT0001 complete [ 5223.622323] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 5225.879945] Key type lgssc unregistered [ 5226.114858] LNet: 163347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5226.119614] LNetError: 163347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5226.134729] LNet: Removed LNI 192.168.202.105@tcp [ 5226.739093] Key type .llcrypt unregistered [ 5226.740751] Key type ._llcrypt unregistered [ 5248.012034] Key type ._llcrypt registered [ 5248.015446] Key type .llcrypt registered [ 5248.150241] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5248.824190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5248.853374] alg: No test for adler32 (adler32-zlib) [ 5249.901711] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 5250.065544] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 5251.719204] Key type lgssc registered [ 5252.663747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5259.793448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5259.830243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5261.420075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5265.075175] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5271.036091] LustreError: 164758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5271.075811] LustreError: 164758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5274.719027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5275.082957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5281.013815] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5290.950732] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5296.766732] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5303.583106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5304.107902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5306.201299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5312.521728] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5321.156264] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5327.955631] Lustre: Failing over lustre-MDT0000 [ 5328.316231] LustreError: 164752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.688409] Lustre: server umount lustre-MDT0000 complete [ 5330.944747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5330.946443] LustreError: 164753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5330.966779] Lustre: Skipped 1 previous similar message [ 5331.003594] LustreError: 164753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5336.037102] LustreError: 164758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5336.056502] LustreError: 164758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5341.155280] LustreError: 164753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5341.192504] LustreError: 164753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5341.669734] LustreError: 164738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.5@tcp arrived at 1788645861 with bad export cookie 11782247501157573233 [ 5347.366075] Lustre: 164069:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788645850/real 1788645850] req@00000000240f3156 x1875531025903232/t0(0) o400->MGC192.168.202.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788645866 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5347.436321] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5351.664637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5352.930941] LustreError: 166668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5352.963694] LustreError: 166668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5357.041250] LustreError: 164066:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a91ccd61 x1875531025908608/t0(0) o250->MGC192.168.202.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5357.532975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5357.606261] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5362.663568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5362.665883] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5364.025360] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5366.462978] Lustre: lustre-MDT0000: Denying connection for new client 0b8aa972-188c-4c11-bc6b-a44c93e0125a (at 192.168.202.5@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5376.993894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5377.014618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5377.024133] Lustre: Skipped 1 previous similar message [ 5377.033533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5378.021929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5378.024106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5380.815570] Lustre: server umount lustre-OST0000 complete [ 5385.925870] LustreError: 168489:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5385.932771] Lustre: 167837:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5385.939561] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5385.961472] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5385.980258] LustreError: 167837:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001650a987 x1875531025924096/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5386.002465] LustreError: 167837:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5386.013889] LustreError: 167837:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5386.024369] Lustre: lustre-MDT0000: Recovery over after 0:24, of 2 clients 0 recovered and 2 were evicted. [ 5386.585331] Lustre: server umount lustre-MDT0000 complete [ 5388.258706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5388.283381] LustreError: 164753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5388.301760] LustreError: 164753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5391.149719] LustreError: 164739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645910 with bad export cookie 11782247501157573331 [ 5391.180772] LustreError: 164739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5391.194456] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5391.698639] Lustre: server umount lustre-MDT0001 complete [ 5403.441337] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 5406.654907] Key type lgssc unregistered [ 5406.993875] LNet: 169241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5406.997363] LNetError: 169241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5407.028199] LNet: Removed LNI 192.168.202.105@tcp [ 5408.114307] Key type .llcrypt unregistered [ 5408.121917] Key type ._llcrypt unregistered [ 5432.105427] Key type ._llcrypt registered [ 5432.107540] Key type .llcrypt registered [ 5432.205110] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5433.181894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5433.204548] alg: No test for adler32 (adler32-zlib) [ 5434.401393] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 5434.773453] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 5436.471504] Key type lgssc registered [ 5438.152978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5445.703263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5445.735542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5447.182592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.043774] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5457.379934] LustreError: 170651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5457.405503] LustreError: 170651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5462.035401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5462.342839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5466.802629] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5475.200874] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5483.137611] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5494.534951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5495.317414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5503.145940] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5511.179290] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5516.806268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5527.020687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5527.021926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5527.033876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5527.042364] Lustre: Skipped 1 previous similar message [ 5528.435114] Lustre: server umount lustre-OST0000 complete [ 5534.185299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5534.193303] LustreError: Skipped 1 previous similar message [ 5534.197181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5534.228252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5534.231066] Lustre: Skipped 1 previous similar message [ 5537.274802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5537.275933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5539.475448] Lustre: server umount lustre-MDT0000 complete [ 5542.379858] LustreError: 170652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5542.408782] LustreError: 170652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5543.773160] LustreError: 170632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646063 with bad export cookie 12094132938799658321 [ 5543.788803] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5544.314453] Lustre: server umount lustre-MDT0001 complete [ 5557.288849] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 5560.006983] Key type lgssc unregistered [ 5560.269768] LNet: 174181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5560.274920] LNetError: 174181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5560.295068] LNet: Removed LNI 192.168.202.105@tcp [ 5561.165380] Key type .llcrypt unregistered [ 5561.167348] Key type ._llcrypt unregistered [ 5591.373098] Key type ._llcrypt registered [ 5591.376294] Key type .llcrypt registered [ 5591.469208] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5592.747539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5592.800939] alg: No test for adler32 (adler32-zlib) [ 5593.913747] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 5594.224717] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 5595.904087] Key type lgssc registered [ 5596.918937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5604.745456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5604.771927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.312508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5610.404529] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5615.597853] LustreError: 175884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.614571] LustreError: 175884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5618.210480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.560668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5623.608362] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5631.985334] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5638.300364] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5645.230254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5645.715852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.835693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5652.408452] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5659.652830] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5672.934041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5672.945224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5672.975312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5673.446726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5673.452748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.236215] Lustre: server umount lustre-OST0000 complete [ 5683.689425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5683.692543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5683.708943] Lustre: Skipped 1 previous similar message [ 5683.718907] Lustre: Skipped 2 previous similar messages [ 5688.058513] Lustre: server umount lustre-MDT0000 complete [ 5688.808708] LustreError: 175884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.837872] LustreError: 175884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5691.327961] LustreError: 175866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646210 with bad export cookie 15481656952378973764 [ 5691.332562] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5691.344135] LustreError: 175866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5691.754413] Lustre: server umount lustre-MDT0001 complete [ 5700.897964] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 5703.944656] Key type lgssc unregistered [ 5704.285685] LNet: 179434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5704.301610] LNetError: 179434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5705.326566] LNet: Removed LNI 192.168.202.105@tcp [ 5706.371760] Key type .llcrypt unregistered [ 5706.374755] Key type ._llcrypt unregistered [ 5734.580736] Key type ._llcrypt registered [ 5734.583626] Key type .llcrypt registered [ 5734.774651] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5736.048509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5736.244087] alg: No test for adler32 (adler32-zlib) [ 5737.400841] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 5737.642332] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 5739.391551] Key type lgssc registered [ 5741.309307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5754.212492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5754.256224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5755.747748] Lustre: *** cfs_fail_loc=135, val=0*** [ 5755.865437] LustreError: 180955:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5755.872837] LustreError: 180955:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.105@tcp: cfg command failed: rc = -2 [ 5755.883936] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5755.883936] [ 5755.895320] LustreError: MGC192.168.202.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5755.911651] LustreError: 180917:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5755.920419] LustreError: 180917:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5755.935706] LustreError: 180917:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5755.941660] LustreError: 180917:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5755.997419] Lustre: server umount lustre-MDT0000 complete [ 5756.002661] LustreError: 180917:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5763.884405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5764.385933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5768.899800] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5773.817779] LustreError: 181327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.850793] LustreError: 181327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5778.239967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.792171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5786.473631] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5794.496623] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5801.768870] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5804.513361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5804.522687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.544588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5808.609907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5808.615696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5810.664884] Lustre: server umount lustre-MDT0000 complete [ 5813.733837] LustreError: 181328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5813.760078] LustreError: 181328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5814.199193] LustreError: 181307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646333 with bad export cookie 10921918688512815600 [ 5814.204924] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5814.206700] LustreError: 181307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5814.535600] Lustre: server umount lustre-MDT0001 complete [ 5828.155878] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 5831.924745] Key type lgssc unregistered [ 5832.372820] LNet: 183646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5832.391785] LNetError: 183646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5832.418952] LNet: Removed LNI 192.168.202.105@tcp [ 5833.322475] Key type .llcrypt unregistered [ 5833.325377] Key type ._llcrypt unregistered [ 5861.940772] Key type ._llcrypt registered [ 5861.945764] Key type .llcrypt registered [ 5862.066786] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 5862.498407] Key type .llcrypt unregistered [ 5862.500991] Key type ._llcrypt unregistered [ 5881.218538] Key type ._llcrypt registered [ 5881.220483] Key type .llcrypt registered [ 5881.338040] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5882.620373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5882.674111] alg: No test for adler32 (adler32-zlib) [ 5883.749135] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 5883.968257] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 5885.720364] Key type lgssc registered [ 5887.279706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5897.168431] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5897.199429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5912.927424] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5918.048262] LustreError: 185773:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 5944.864533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.025828] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5962.702586] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5970.588555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.225220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.325819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5975.976889] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5976.563096] LustreError: 185799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5976.596178] LustreError: 185799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5979.620313] LustreError: 186813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5979.638275] LustreError: 186813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5984.739806] LustreError: 187249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5984.758352] LustreError: 187249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5985.898241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5986.358540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5992.743151] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6001.186789] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6006.913361] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6009.972266] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6022.119825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6022.125727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6022.130703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6022.168612] Lustre: Skipped 1 previous similar message [ 6026.914880] Lustre: server umount lustre-OST0000 complete [ 6032.355285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6032.358917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6032.393563] Lustre: Skipped 1 previous similar message [ 6032.414651] Lustre: Skipped 2 previous similar messages [ 6037.472981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6037.480304] Lustre: Skipped 1 previous similar message [ 6037.848178] Lustre: server umount lustre-MDT0000 complete [ 6041.572663] LustreError: 186818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6041.620918] LustreError: 186818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6042.162967] LustreError: 188435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646561 with bad export cookie 18129845164664362556 [ 6042.164576] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6042.173449] LustreError: 188435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6042.667344] Lustre: server umount lustre-MDT0001 complete [ 6054.967136] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6058.258975] Key type lgssc unregistered [ 6058.591063] LNet: 189738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6058.599078] LNetError: 189738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6058.620258] LNet: Removed LNI 192.168.202.105@tcp [ 6059.558616] Key type .llcrypt unregistered [ 6059.560601] Key type ._llcrypt unregistered [ 6082.423429] Key type ._llcrypt registered [ 6082.424864] Key type .llcrypt registered [ 6082.502341] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6083.644552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6083.668499] alg: No test for adler32 (adler32-zlib) [ 6084.792926] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 6085.073911] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6086.735183] Key type lgssc registered [ 6087.780458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6096.045357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6096.072896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6097.748707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.710564] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6108.142872] LustreError: 191146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.179393] LustreError: 191146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6110.784740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.073875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6116.057439] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6124.106678] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6130.315886] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6137.809035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6138.484339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.599821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6145.404740] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6155.650566] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6162.636793] Lustre: Failing over lustre-MDT0000 [ 6163.291573] Lustre: server umount lustre-MDT0000 complete [ 6163.894919] LustreError: 191142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6166.498649] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6166.506135] LustreError: 191145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6166.537612] Lustre: Skipped 1 previous similar message [ 6168.984878] LustreError: 191142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6169.022387] LustreError: 191142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6174.135823] LustreError: 193454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6174.157297] LustreError: 193454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6182.778340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6182.880960] Lustre: 190457:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788646686/real 1788646686] req@000000008191eb98 x1875531901464192/t0(0) o400->MGC192.168.202.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788646702 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6182.919297] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6182.942025] LustreError: 191146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.976712] LustreError: 191146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6193.079814] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa601779ee2816dd4 [ 6193.092665] Lustre: MGC192.168.202.105@tcp: Connection restored to 0@lo (at 0@lo) [ 6193.409964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6193.478514] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6194.584726] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6197.853927] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6198.770917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6198.810943] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6198.879465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6202.152903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6204.149229] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6209.506087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6209.519993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6209.540283] Lustre: Skipped 1 previous similar message [ 6209.544820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6211.730741] Lustre: server umount lustre-OST0000 complete [ 6219.232627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6219.235524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6219.249449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6219.267427] Lustre: Skipped 1 previous similar message [ 6221.765740] Lustre: server umount lustre-MDT0000 complete [ 6224.352879] LustreError: 191146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6224.366632] LustreError: 191146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6224.859197] LustreError: 191125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646744 with bad export cookie 11961973609561419220 [ 6224.874445] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6235.253608] Lustre: server umount lustre-MDT0001 complete [ 6245.971573] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6248.583728] Key type lgssc unregistered [ 6248.882056] LNet: 195886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6248.893506] LNetError: 195886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6248.910294] LNet: Removed LNI 192.168.202.105@tcp [ 6249.816516] Key type .llcrypt unregistered [ 6249.818527] Key type ._llcrypt unregistered [ 6279.897562] Key type ._llcrypt registered [ 6279.904377] Key type .llcrypt registered [ 6280.085823] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6281.697038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6281.737762] alg: No test for adler32 (adler32-zlib) [ 6282.803902] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 6283.039956] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6284.735243] Key type lgssc registered [ 6286.329136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6294.918898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6294.936443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6296.395386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6301.121656] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6305.778292] LustreError: 197297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.798295] LustreError: 197297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6310.544426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6310.878129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6315.250061] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6322.239354] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6328.213975] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6335.087807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6335.440796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6340.593638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6341.766506] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6349.175989] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6353.153828] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6358.490629] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6372.788328] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6381.540603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.575729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6381.589921] Lustre: Skipped 1 previous similar message [ 6381.610745] Lustre: Skipped 1 previous similar message [ 6383.930916] Lustre: server umount lustre-OST0000 complete [ 6387.688502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6387.705325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.722288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6391.782946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6391.786155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6394.008926] Lustre: server umount lustre-MDT0000 complete [ 6396.900475] LustreError: 197297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6396.926803] LustreError: 197297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6397.354905] LustreError: 197277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646916 with bad export cookie 13998099133827830829 [ 6397.357546] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6397.387832] LustreError: 197277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6397.879687] Lustre: server umount lustre-MDT0001 complete [ 6410.840540] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6413.540653] Key type lgssc unregistered [ 6413.787220] LNet: 201140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6413.792603] LNetError: 201140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6413.815461] LNet: Removed LNI 192.168.202.105@tcp [ 6414.713592] Key type .llcrypt unregistered [ 6414.716646] Key type ._llcrypt unregistered [ 6432.187264] Key type ._llcrypt registered [ 6432.190139] Key type .llcrypt registered [ 6432.339495] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6433.677738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6433.778797] alg: No test for adler32 (adler32-zlib) [ 6434.875258] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 6435.181138] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6436.903268] Key type lgssc registered [ 6438.268091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6445.963796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6445.981354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6447.442450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6451.785093] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6456.839930] LustreError: 202406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6456.861433] LustreError: 202406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6460.390121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.921905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6465.342617] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6472.106533] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6477.605811] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6484.594827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6485.012619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6491.332861] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6492.219681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6498.777713] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6504.928326] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6514.352383] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6519.926219] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6528.938859] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6537.185194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6537.195140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6537.210028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6538.212307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6538.215532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6538.238176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6541.472928] Lustre: server umount lustre-OST0000 complete [ 6547.942217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6547.957474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6547.979568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6547.990307] Lustre: Skipped 1 previous similar message [ 6552.517186] Lustre: server umount lustre-MDT0000 complete [ 6553.569069] LustreError: 202406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6553.592782] LustreError: 202406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6556.168494] LustreError: 206081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647075 with bad export cookie 5443765405773089531 [ 6556.177951] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6556.187575] LustreError: 206081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6557.041881] Lustre: server umount lustre-MDT0001 complete [ 6567.036600] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6569.684845] Key type lgssc unregistered [ 6570.051899] LNet: 207232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6570.060517] LNetError: 207232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6570.079862] LNet: Removed LNI 192.168.202.105@tcp [ 6570.878992] Key type .llcrypt unregistered [ 6570.882502] Key type ._llcrypt unregistered [ 6595.744561] Key type ._llcrypt registered [ 6595.748041] Key type .llcrypt registered [ 6595.885000] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6596.733643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6596.917783] alg: No test for adler32 (adler32-zlib) [ 6597.982977] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 6598.290859] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6600.056359] Key type lgssc registered [ 6601.493689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6609.836179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6609.856350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.435873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6615.504596] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6620.649435] LustreError: 208645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6620.671661] LustreError: 208645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6623.580215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6623.995929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6628.406393] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6635.113216] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6639.801599] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6645.887484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6646.454279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6652.502310] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6655.706051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6660.142571] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6676.452597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.454877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6676.474124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6681.579685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6681.593557] Lustre: Skipped 2 previous similar messages [ 6682.659747] Lustre: server umount lustre-OST0000 complete [ 6686.692170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6686.700121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6686.738255] Lustre: Skipped 2 previous similar messages [ 6686.770250] Lustre: Skipped 1 previous similar message [ 6691.809162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6691.822158] Lustre: Skipped 1 previous similar message [ 6692.729805] Lustre: server umount lustre-MDT0000 complete [ 6696.639968] LustreError: 208625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647216 with bad export cookie 316458069844153877 [ 6696.641261] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6696.656115] LustreError: 208625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6697.017775] Lustre: server umount lustre-MDT0001 complete [ 6713.237692] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6716.348853] Key type lgssc unregistered [ 6716.693914] LNet: 212328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6716.702099] LNetError: 212328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6716.726140] LNet: Removed LNI 192.168.202.105@tcp [ 6717.681979] Key type .llcrypt unregistered [ 6717.684158] Key type ._llcrypt unregistered [ 6736.591448] Key type ._llcrypt registered [ 6736.593832] Key type .llcrypt registered [ 6736.705960] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6737.999682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6738.035663] alg: No test for adler32 (adler32-zlib) [ 6739.141133] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 6739.361733] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6741.063390] Key type lgssc registered [ 6742.095305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6749.096937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6749.112776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6750.617027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6755.847707] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6760.929433] LustreError: 213591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6760.951101] LustreError: 213591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6766.061277] LustreError: 213590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6766.277538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6766.871215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6771.277438] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6778.618722] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6784.458288] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6791.679214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6792.067742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6796.147809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6797.048381] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6804.417809] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6821.875472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6821.877263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6821.884342] LustreError: Skipped 1 previous similar message [ 6821.888185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6821.901818] Lustre: Skipped 1 previous similar message [ 6821.915439] Lustre: Skipped 1 previous similar message [ 6823.333384] Lustre: server umount lustre-OST0000 complete [ 6828.512982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6828.523057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6828.537384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6832.096610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6832.098234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6832.116839] Lustre: Skipped 1 previous similar message [ 6833.407691] Lustre: server umount lustre-MDT0000 complete [ 6837.136583] LustreError: 213571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647356 with bad export cookie 4874788662648954483 [ 6837.137984] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6837.149058] LustreError: 213571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6837.461288] Lustre: server umount lustre-MDT0001 complete [ 6849.527784] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6853.398337] Key type lgssc unregistered [ 6853.773043] LNet: 217271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6853.787880] LNetError: 217271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6853.809569] LNet: Removed LNI 192.168.202.105@tcp [ 6854.726949] Key type .llcrypt unregistered [ 6854.730778] Key type ._llcrypt unregistered [ 6877.760583] Key type ._llcrypt registered [ 6877.763660] Key type .llcrypt registered [ 6877.863271] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6879.096631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6879.217680] alg: No test for adler32 (adler32-zlib) [ 6880.254211] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 6880.449886] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6882.079223] Key type lgssc registered [ 6883.249710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6891.507369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6891.541407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6893.110997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6897.806517] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6903.270182] LustreError: 218533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6903.291537] LustreError: 218533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6906.620786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6907.076522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6912.178286] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6920.581511] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6926.469833] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6933.873152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6934.593790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6937.668771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6941.820729] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6953.034409] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6962.816710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6962.915166] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6963.122648] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6969.828561] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6976.940855] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6978.551975] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6990.519372] Lustre: Permanently deactivating lustre-OST0001 [ 6990.546356] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6997.082207] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7007.155957] Lustre: Permanently reactivating lustre-OST0001 [ 7007.181073] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7007.187868] Lustre: Skipped 2 previous similar messages [ 7013.989437] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7014.006362] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7014.011783] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7014.019606] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7014.062252] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7018.980068] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7019.014919] Lustre: Skipped 1 previous similar message [ 7019.024455] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7019.029550] Lustre: Skipped 1 previous similar message [ 7022.843103] Lustre: server umount lustre-OST0001 complete [ 7024.107986] LustreError: 220985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7024.156699] LustreError: 220985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7028.705320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7028.710202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7028.716517] Lustre: Skipped 1 previous similar message [ 7028.720732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7029.236562] LustreError: 220985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7029.268561] LustreError: 220985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7033.521691] Lustre: server umount lustre-OST0000 complete [ 7039.456505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7039.463080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7039.480912] Lustre: Skipped 2 previous similar messages [ 7039.496681] Lustre: Skipped 3 previous similar messages [ 7044.576219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.607814] LustreError: 218528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7044.809389] Lustre: server umount lustre-MDT0000 complete [ 7049.221139] LustreError: 218513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647568 with bad export cookie 6841053759645174812 [ 7049.231596] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7049.243946] LustreError: 218513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7049.622840] Lustre: server umount lustre-MDT0001 complete [ 7060.788354] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7063.354645] Key type lgssc unregistered [ 7063.638481] LNet: 223603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7063.642364] LNetError: 223603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7063.659351] LNet: Removed LNI 192.168.202.105@tcp [ 7064.900045] Key type .llcrypt unregistered [ 7064.902341] Key type ._llcrypt unregistered [ 7106.088610] Key type ._llcrypt registered [ 7106.095354] Key type .llcrypt registered [ 7106.199689] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7107.733603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7107.796287] alg: No test for adler32 (adler32-zlib) [ 7108.914406] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 7109.437517] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7111.207162] Key type lgssc registered [ 7113.059148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7121.435523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7121.463268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7122.923268] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7123.012537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7123.306585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7128.123290] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7136.981711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.133733] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7137.159569] Lustre: 225805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7137.371539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7142.623435] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7150.541933] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7157.118542] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7164.059880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7164.205051] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7164.353533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7171.159464] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7173.765743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7180.312885] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7184.468925] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7189.174758] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7195.264896] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7203.516000] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7213.597270] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7236.348740] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7236.359544] Lustre: Skipped 2 previous similar messages [ 7271.791759] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7271.802193] Lustre: Skipped 4 previous similar messages [ 7336.319460] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7336.328251] Lustre: Skipped 7 previous similar messages [ 7353.313806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7353.323345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7353.340169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7353.365388] Lustre: Skipped 1 previous similar message [ 7358.435335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7358.440338] Lustre: Skipped 2 previous similar messages [ 7359.171513] Lustre: server umount lustre-OST0000 complete [ 7363.554656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7363.568605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7363.571468] Lustre: Skipped 1 previous similar message [ 7368.674508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7368.694622] Lustre: Skipped 1 previous similar message [ 7373.794258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7377.375254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7377.702259] Lustre: server umount lustre-MDT0000 complete [ 7382.130895] LustreError: 228373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647901 with bad export cookie 16985432705410637203 [ 7382.132103] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7382.139805] LustreError: 228373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7382.431595] Lustre: server umount lustre-MDT0001 complete [ 7396.338358] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7400.162526] Key type lgssc unregistered [ 7400.452760] LNet: 230607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7400.465026] LNetError: 230607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7400.482881] LNet: Removed LNI 192.168.202.105@tcp [ 7401.403799] Key type .llcrypt unregistered [ 7401.405914] Key type ._llcrypt unregistered [ 7436.221985] Key type ._llcrypt registered [ 7436.227793] Key type .llcrypt registered [ 7436.340245] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7437.924082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7437.982317] alg: No test for adler32 (adler32-zlib) [ 7439.108733] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 7439.403141] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7441.071831] Key type lgssc registered [ 7442.166865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7450.415914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7450.461679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.065675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7456.882315] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7462.370633] LustreError: 232021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7462.391514] LustreError: 232021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7466.788743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7467.522141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7474.140132] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7483.593940] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7489.000058] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7496.524089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7496.947521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7497.986204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7503.478874] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7511.298102] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7527.905558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7527.910936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7527.938453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7527.957646] Lustre: Skipped 1 previous similar message [ 7530.036098] Lustre: server umount lustre-OST0000 complete [ 7534.048300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7534.056109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7534.079542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7534.087875] Lustre: Skipped 1 previous similar message [ 7538.149752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7538.156473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7538.250795] Lustre: Skipped 1 previous similar message [ 7543.264565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7543.274351] Lustre: Skipped 1 previous similar message [ 7548.388494] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7548.410678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7548.433906] Lustre: Skipped 1 previous similar message [ 7548.829784] Lustre: server umount lustre-MDT0000 complete [ 7553.510766] LustreError: 232749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7555.932094] LustreError: 232756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648075 with bad export cookie 14782396193079189348 [ 7555.938277] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7555.954064] LustreError: 232756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7556.476119] Lustre: server umount lustre-MDT0001 complete [ 7569.269207] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7572.476298] Key type lgssc unregistered [ 7572.842312] LNet: 235711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7572.853709] LNetError: 235711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7572.887493] LNet: Removed LNI 192.168.202.105@tcp [ 7573.757186] Key type .llcrypt unregistered [ 7573.760442] Key type ._llcrypt unregistered [ 7604.388114] Key type ._llcrypt registered [ 7604.389568] Key type .llcrypt registered [ 7604.531770] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7605.832465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7605.882986] alg: No test for adler32 (adler32-zlib) [ 7607.845830] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 7608.304834] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7610.191265] Key type lgssc registered [ 7611.700868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7881.649430] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7884.384976] Key type lgssc unregistered [ 7884.637436] LNet: 237730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7884.649515] LNetError: 237730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7884.680064] LNet: Removed LNI 192.168.202.105@tcp [ 7885.449503] Key type .llcrypt unregistered [ 7885.457437] Key type ._llcrypt unregistered [ 7908.349571] Key type ._llcrypt registered [ 7908.355778] Key type .llcrypt registered [ 7908.530630] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 7923.033582] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7924.072844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7924.107401] alg: No test for adler32 (adler32-zlib) [ 7925.129752] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 7925.373364] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7927.031179] Key type lgssc registered [ 7928.116930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7955.291357] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7608 sec ========== 18:47:53 (1788648473) [ 7956.756799] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:47:55 (1788648475) === [ 7960.843701] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:47:58 (1788648478) === [ 7996.747707] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7998.816882] Key type lgssc unregistered [ 7999.059688] LNet: 242965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7999.062422] LNetError: 242965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7999.070602] LNet: Removed LNI 192.168.202.105@tcp [ 7999.769815] Key type .llcrypt unregistered [ 7999.772538] Key type ._llcrypt unregistered