[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 520813232 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002380] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007620] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010131] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.013079] *** VALIDATE selinux *** [ 0.022053] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027263] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030153] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031108] *** VALIDATE tmpfs *** [ 0.032453] *** VALIDATE proc *** [ 0.033267] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.036085] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040032] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.043688] debug: unmapping init [mem 0xffffffff92059000-0xffffffff92060fff] [ 0.045249] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046697] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047035] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054352] rcu: Hierarchical SRCU implementation. [ 0.056516] smp: Bringing up secondary CPUs ... [ 0.057571] x86: Booting SMP configuration: [ 0.058024] .... node #0, CPUs: #1 #2 #3 [ 0.061341] smp: Brought up 1 node, 4 CPUs [ 0.063010] smpboot: Max logical packages: 1 [ 0.064016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152000] node 0 deferred pages initialised in 86ms [ 0.155357] devtmpfs: initialized [ 0.157275] x86/mm: Memory block size: 128MB [ 0.159777] gcov: version magic: 0x41383552 [ 0.163293] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.164089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165259] pinctrl core: initialized pinctrl subsystem [ 0.166206] [ 0.166892] ************************************************************* [ 0.167016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168012] ** ** [ 0.169014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170013] ** ** [ 0.171011] ** This means that this kernel is built to expose internal ** [ 0.172015] ** IOMMU data structures, which may compromise security on ** [ 0.173015] ** your system. ** [ 0.174015] ** ** [ 0.175012] ** If you see this message and you are not debugging the ** [ 0.176012] ** kernel, report this immediately to your vendor! ** [ 0.177016] ** ** [ 0.178013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179013] ************************************************************* [ 0.180573] NET: Registered protocol family 16 [ 0.181341] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.184540] cpuidle: using governor menu [ 0.186778] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.190517] PCI: Using configuration type 1 for base access [ 0.193122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203060] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206044] cryptd: max_cpu_qlen set to 1000 [ 0.208245] ACPI: Added _OSI(Module Device) [ 0.210018] ACPI: Added _OSI(Processor Device) [ 0.211032] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218490] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226712] ACPI: Interpreter enabled [ 0.228064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230015] ACPI: Using IOAPIC for interrupt routing [ 0.232127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.236640] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.249491] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.252039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.258075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.263504] acpiphp: Slot [2] registered [ 0.265276] acpiphp: Slot [5] registered [ 0.267250] acpiphp: Slot [6] registered [ 0.269213] acpiphp: Slot [7] registered [ 0.271302] acpiphp: Slot [8] registered [ 0.273180] acpiphp: Slot [9] registered [ 0.275263] acpiphp: Slot [10] registered [ 0.277291] acpiphp: Slot [3] registered [ 0.279141] acpiphp: Slot [4] registered [ 0.280141] acpiphp: Slot [11] registered [ 0.282220] acpiphp: Slot [12] registered [ 0.284139] acpiphp: Slot [13] registered [ 0.286378] acpiphp: Slot [14] registered [ 0.289196] acpiphp: Slot [15] registered [ 0.290453] acpiphp: Slot [16] registered [ 0.293145] acpiphp: Slot [17] registered [ 0.294189] acpiphp: Slot [18] registered [ 0.296230] acpiphp: Slot [19] registered [ 0.298115] acpiphp: Slot [20] registered [ 0.300114] acpiphp: Slot [21] registered [ 0.302104] acpiphp: Slot [22] registered [ 0.303120] acpiphp: Slot [23] registered [ 0.305119] acpiphp: Slot [24] registered [ 0.306116] acpiphp: Slot [25] registered [ 0.308118] acpiphp: Slot [26] registered [ 0.310163] acpiphp: Slot [27] registered [ 0.312100] acpiphp: Slot [28] registered [ 0.313293] acpiphp: Slot [29] registered [ 0.315139] acpiphp: Slot [30] registered [ 0.317121] acpiphp: Slot [31] registered [ 0.318057] PCI host bridge to bus 0000:00 [ 0.320023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.323023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326040] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.329026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.337279] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341115] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345529] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.362551] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.375480] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378931] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.385791] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.406026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.412015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.417371] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.427015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.435017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.452015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.462369] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.469012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.498015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.507727] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.516015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.524016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.549016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.562346] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.569017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.581013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.596016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.606448] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.618024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.628016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.655015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.668630] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.680014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.692015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.713016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.724283] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.726407] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.729394] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.732886] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.735263] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.739145] iommu: Default domain type: Passthrough [ 0.741396] SCSI subsystem initialized [ 0.742123] ACPI: bus type USB registered [ 0.743092] usbcore: registered new interface driver usbfs [ 0.746118] usbcore: registered new interface driver hub [ 0.748085] usbcore: registered new device driver usb [ 0.749179] pps_core: LinuxPPS API ver. 1 registered [ 0.751015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.755050] PTP clock support registered [ 0.756149] EDAC MC: Ver: 3.0.0 [ 0.758144] PCI: Using ACPI for IRQ routing [ 0.759990] NetLabel: Initializing [ 0.761010] NetLabel: domain hash size = 128 [ 0.763011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.765071] NetLabel: unlabeled traffic allowed by default [ 0.767137] vgaarb: loaded [ 0.770072] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.771012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.781115] clocksource: Switched to clocksource kvm-clock [ 0.889391] VFS: Disk quotas dquot_6.6.0 [ 0.891111] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.894145] *** VALIDATE ramfs *** [ 0.895723] *** VALIDATE hugetlbfs *** [ 0.897296] pnp: PnP ACPI init [ 0.900031] pnp: PnP ACPI: found 6 devices [ 0.916970] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.920579] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.922899] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.925110] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.927581] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.930316] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.933113] NET: Registered protocol family 2 [ 0.935767] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.941031] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.945660] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.950475] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.953938] TCP: Hash tables configured (established 65536 bind 65536) [ 0.956819] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.960107] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.963161] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.965324] NET: Registered protocol family 1 [ 0.967990] RPC: Registered named UNIX socket transport module. [ 0.970125] RPC: Registered udp transport module. [ 0.971962] RPC: Registered tcp transport module. [ 0.973864] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.976923] NET: Registered protocol family 44 [ 0.978866] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.981514] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.983924] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.986297] PCI: CLS 0 bytes, default 64 [ 0.988109] Unpacking initramfs... [ 2.409801] debug: unmapping init [mem 0xffff8fed7cc54000-0xffff8fed7ffbffff] [ 2.414889] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.418076] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.421817] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.929477] Initialise system trusted keyrings [ 2.931655] Key type blacklist registered [ 2.933487] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.945976] zbud: loaded [ 2.950435] *** VALIDATE nfs *** [ 2.951589] *** VALIDATE nfs4 *** [ 2.953321] pstore: using deflate compression [ 2.956319] Platform Keyring initialized [ 3.071368] NET: Registered protocol family 38 [ 3.073896] Key type asymmetric registered [ 3.075548] Asymmetric key parser 'x509' registered [ 3.080463] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.083515] io scheduler mq-deadline registered [ 3.085227] io scheduler kyber registered [ 3.086751] io scheduler bfq registered [ 3.088601] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.091700] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.094587] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.097363] ACPI: Power Button [PWRF] [ 3.102586] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.110056] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.120929] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.127818] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.142741] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.172107] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.200122] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.206408] Non-volatile memory driver v1.3 [ 3.208723] Linux agpgart interface v0.103 [ 3.243523] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.246472] vda: detected capacity change from 0 to 74698752 [ 3.268569] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.272265] vdb: detected capacity change from 0 to 1073741824 [ 3.292884] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.296230] vdc: detected capacity change from 0 to 2621440000 [ 3.311309] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.314670] vdd: detected capacity change from 0 to 2621440000 [ 3.329059] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.332150] vde: detected capacity change from 0 to 4294967296 [ 3.346519] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.349829] vdf: detected capacity change from 0 to 4294967296 [ 3.357307] libphy: Fixed MDIO Bus: probed [ 3.362342] usbcore: registered new interface driver usbserial_generic [ 3.365308] usbserial: USB Serial support registered for generic [ 3.367856] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.372261] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.374055] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.376794] mousedev: PS/2 mouse device common for all mice [ 3.380081] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.381385] rtc_cmos 00:05: RTC can wake from S4 [ 3.388049] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.389167] rtc_cmos 00:05: registered as rtc0 [ 3.393648] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.397138] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.397211] intel_pstate: CPU model not supported [ 3.404086] hid: raw HID events driver (C) Jiri Kosina [ 3.409670] usbcore: registered new interface driver usbhid [ 3.413087] usbhid: USB HID core driver [ 3.414961] drop_monitor: Initializing network drop monitor service [ 3.418161] Initializing XFRM netlink socket [ 3.420493] NET: Registered protocol family 10 [ 3.423474] Segment Routing with IPv6 [ 3.424495] NET: Registered protocol family 17 [ 3.425815] mpls_gso: MPLS GSO support [ 3.432404] RAS: Correctable Errors collector initialized. [ 3.434774] AVX version of gcm_enc/dec engaged. [ 3.436600] AES CTR mode by8 optimization enabled [ 3.515346] sched_clock: Marking stable (3515269518, 0)->(4482771697, -967502179) [ 3.519225] registered taskstats version 1 [ 3.521143] Loading compiled-in X.509 certificates [ 3.522990] zswap: loaded using pool lzo/zbud [ 3.547059] Key type big_key registered [ 3.558868] Key type encrypted registered [ 3.561184] ima: No TPM chip found, activating TPM-bypass! [ 3.563235] ima: Allocated hash algorithm: sha1 [ 3.565070] ima: No architecture policies found [ 3.566913] evm: Initialising EVM extended attributes: [ 3.568829] evm: security.selinux [ 3.570258] evm: security.ima [ 3.571248] evm: security.capability [ 3.572578] evm: HMAC attrs: 0x1 [ 3.575182] rtc_cmos 00:05: setting system clock to 2026-09-03 13:00:05 UTC (1788440405) [ 3.581970] debug: unmapping init [mem 0xffffffff93003000-0xffffffff931fffff] [ 3.584975] debug: unmapping init [mem 0xffffffff91d82000-0xffffffff92058fff] [ 3.592156] Write protecting the kernel read-only data: 28672k [ 3.596228] debug: unmapping init [mem 0xffffffff90403000-0xffffffff905fffff] [ 3.599618] debug: unmapping init [mem 0xffffffff90d14000-0xffffffff90dfffff] [ 3.634761] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.642220] systemd[1]: Detected virtualization kvm. [ 3.643904] systemd[1]: Detected architecture x86-64. [ 3.645723] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.670989] systemd[1]: No hostname configured. [ 3.672780] systemd[1]: Set hostname to . [ 3.674668] random: systemd: uninitialized urandom read (16 bytes read) [ 3.676725] systemd[1]: Initializing machine ID from random generator. [ 3.717085] random: ln: uninitialized urandom read (6 bytes read) [ 3.834844] random: systemd: uninitialized urandom read (16 bytes read) [ 3.838317] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.847859] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 3.853581] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.539778] device-mapper: uevent: version 1.0.3 [ 4.542398] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.281176] virtio_net virtio0 ens2: renamed from eth0 [ 5.282426] random: fast init done [ 5.344041] scsi host0: ata_piix [ 5.370935] scsi host1: ata_piix [ 5.372677] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.375081] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.887090] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.077540] random: crng init done [ 10.078849] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.447253] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.541178] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.829744] SELinux: Disabled at runtime. [ 11.896496] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.905713] systemd[1]: Detected virtualization kvm. [ 11.907457] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.412370] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.415918] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.420841] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.425209] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.429434] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.437764] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.442635] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [[ 12.497453] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.891026] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.204689] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.230155] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.311288] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.322067] EDAC sbridge: Ver: 1.1.2 [ 15.218237] Key type dns_resolver registered [ 15.520405] NFS: Registering the id_resolver key type [ 15.522638] Key type id_resolver registered [ 15.523655] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Network Manager... [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg649-server login: [ 51.038221] libcfs: loading out-of-tree module taints kernel. [ 51.127825] Key type ._llcrypt registered [ 51.132034] Key type .llcrypt registered [ 51.274198] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 72.811972] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 73.751097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 73.759413] alg: No test for adler32 (adler32-zlib) [ 74.582528] hrtimer: interrupt took 10388829 ns [ 75.288975] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 76.077721] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 77.890282] Key type lgssc registered [ 79.462534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.243272] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 136.012798] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 148.873437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 148.906161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.193825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.236453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.359433] Lustre: lustre-MDT0000: new disk, initializing [ 150.485611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.509978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 155.960882] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.613450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.760338] Lustre: 6523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 170.794928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 170.801133] Lustre: Skipped 1 previous similar message [ 170.891379] Lustre: lustre-MDT0001: new disk, initializing [ 170.978544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.029079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 171.055139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 176.011428] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.723700] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 190.583586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.798697] Lustre: lustre-OST0000: new disk, initializing [ 190.801504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.805140] Lustre: 8463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.872508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.672416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.693538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.811667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.018273] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.658734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.754925] Lustre: lustre-OST0001: new disk, initializing [ 214.760945] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 214.773839] Lustre: 9536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.854883] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.760471] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.318987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 224.347528] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 224.410597] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.984528] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 243.198889] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 249.227443] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing check_logdir /tmp/testlogs/ [ 253.853923] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing yml_node [ 258.282529] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 260.884924] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 263.611381] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 265.414853] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:04:24 EDT 2026 [ 283.878739] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 285.570621] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 304.097718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 304.112277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.149393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.145874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.147456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.169773] Lustre: Skipped 1 previous similar message [ 308.896757] Lustre: server umount lustre-MDT0000 complete [ 312.734256] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440714 with bad export cookie 18392644056439442474 [ 312.740138] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.745943] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 316.385792] LustreError: 11979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.386180] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.410605] LustreError: 11979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 316.431232] Lustre: Skipped 2 previous similar messages [ 316.436377] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.447937] Lustre: Skipped 1 previous similar message [ 317.420706] LustreError: 6531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.448581] LustreError: 6531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 322.529039] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.529064] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.546545] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 322.550120] Lustre: Skipped 3 previous similar messages [ 327.136583] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 327.422724] Lustre: server umount lustre-MDT0001 complete [ 331.633777] Lustre: server umount lustre-OST0000 complete [ 336.021817] Lustre: server umount lustre-OST0001 complete [ 343.487316] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 355.362453] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 396.195913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.430513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 396.453571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 396.532463] Lustre: lustre-MDT0000: new disk, initializing [ 396.613556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 396.627440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.601864] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 412.650208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.749987] Lustre: 15092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 412.796506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 412.799743] Lustre: Skipped 1 previous similar message [ 412.872792] Lustre: lustre-MDT0001: new disk, initializing [ 412.957252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.990322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 412.999580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 417.498610] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 427.566065] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 433.864237] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 441.408943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.644375] Lustre: lustre-OST0000: new disk, initializing [ 441.647823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 441.656413] Lustre: 17059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 441.742210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.631240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 443.643406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 443.768285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 447.629966] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 457.264548] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 462.259514] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 462.425264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 466.742431] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 466.949665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 469.475875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 469.477625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 469.489479] LustreError: Skipped 1 previous similar message [ 469.493169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 469.504317] Lustre: Skipped 2 previous similar messages [ 469.520961] Lustre: Skipped 1 previous similar message [ 479.717157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 479.725811] Lustre: Skipped 3 previous similar messages [ 483.296222] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 483.500378] Lustre: server umount lustre-OST0000 complete [ 489.962714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.979637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 493.788832] Lustre: server umount lustre-MDT0000 complete [ 495.074275] LustreError: 15099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.089207] LustreError: 15099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 497.735280] LustreError: 17058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440899 with bad export cookie 18392644056439444007 [ 497.737517] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.747561] LustreError: 17058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 498.267424] Lustre: server umount lustre-MDT0001 complete [ 502.174642] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:08:21 (1788440901) [ 509.840921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.440437] LustreError: 19325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.566661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.609179] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 520.169755] LustreError: 19325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.178118] LustreError: 19325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 524.461658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.004879] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 537.896160] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 544.168894] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 551.948047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 559.243635] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 567.047986] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 579.051202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 579.060722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.069580] Lustre: Skipped 1 previous similar message [ 579.082661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 579.094281] Lustre: Skipped 2 previous similar messages [ 584.565834] Lustre: server umount lustre-OST0000 complete [ 603.104421] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 603.257201] Lustre: server umount lustre-MDT0000 complete [ 605.667819] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.690668] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 608.154592] LustreError: 19307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441010 with bad export cookie 18392644056439444889 [ 608.161246] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.166250] LustreError: 19307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 608.740305] Lustre: server umount lustre-MDT0001 complete [ 622.235216] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 625.663355] Key type lgssc unregistered [ 626.296705] LNet: 22862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.305957] LNetError: 22862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.351714] LNet: Removed LNI 192.168.206.149@tcp [ 628.055127] Key type .llcrypt unregistered [ 628.057804] Key type ._llcrypt unregistered [ 654.168181] Key type ._llcrypt registered [ 654.170825] Key type .llcrypt registered [ 654.290938] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 656.052444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 656.100751] alg: No test for adler32 (adler32-zlib) [ 657.083413] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 657.254102] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 658.944436] Key type lgssc registered [ 659.976504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 668.615734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 668.643241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.349592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 675.848324] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 680.936918] LustreError: 24279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 680.963817] LustreError: 24279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 686.057621] LustreError: 24280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 687.105400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.357350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 692.276906] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 700.279821] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 705.957263] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 712.578424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 712.859784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.958792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 719.082081] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 727.533578] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 742.881584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 742.889088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.904635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 742.925775] Lustre: Skipped 1 previous similar message [ 748.004103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 748.011915] Lustre: Skipped 2 previous similar messages [ 748.092954] Lustre: server umount lustre-OST0000 complete [ 754.155840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 754.173421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.206851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 756.198512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.219465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 756.234331] Lustre: Skipped 1 previous similar message [ 758.261745] Lustre: server umount lustre-MDT0000 complete [ 761.317396] LustreError: 24275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 761.360284] LustreError: 24275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 761.978754] LustreError: 24261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441163 with bad export cookie 11624151047806739655 [ 761.989208] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.998237] LustreError: 24261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 762.042562] LustreError: 23592:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 762.367775] Lustre: server umount lustre-MDT0001 complete [ 775.993522] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 778.727885] Key type lgssc unregistered [ 779.125914] LNet: 28257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.135543] LNetError: 28257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.169372] LNet: Removed LNI 192.168.206.149@tcp [ 780.020058] Key type .llcrypt unregistered [ 780.022345] Key type ._llcrypt unregistered [ 803.656423] Key type ._llcrypt registered [ 803.658451] Key type .llcrypt registered [ 803.795861] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 804.734698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 804.773366] alg: No test for adler32 (adler32-zlib) [ 805.829616] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 806.090968] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 807.736223] Key type lgssc registered [ 809.602580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 817.061975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 817.074040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.464272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 823.564745] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 828.899168] LustreError: 29671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 828.934930] LustreError: 29671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 832.262905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.616253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 836.746152] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 844.676822] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 850.929771] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 864.236769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 864.563747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.115406] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 876.541963] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 888.202581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 898.536203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 898.545028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 898.545781] LustreError: Skipped 1 previous similar message [ 898.550444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 898.563259] Lustre: Skipped 1 previous similar message [ 903.655922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 903.663243] Lustre: Skipped 1 previous similar message [ 904.045384] Lustre: server umount lustre-OST0000 complete [ 908.773190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.797479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 908.797479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 913.896738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 913.904605] Lustre: Skipped 1 previous similar message [ 914.001752] Lustre: server umount lustre-MDT0000 complete [ 917.747353] LustreError: 31858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441319 with bad export cookie 16170851142350717419 [ 917.760875] LustreError: 31858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 917.762355] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.218978] Lustre: server umount lustre-MDT0001 complete [ 928.921467] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 931.420631] Key type lgssc unregistered [ 931.712646] LNet: 33577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.722868] LNetError: 33577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.786120] LNet: Removed LNI 192.168.206.149@tcp [ 933.692931] Key type .llcrypt unregistered [ 933.696215] Key type ._llcrypt unregistered [ 954.955271] Key type ._llcrypt registered [ 954.958693] Key type .llcrypt registered [ 955.093807] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 955.965251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 956.113077] alg: No test for adler32 (adler32-zlib) [ 957.148383] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 957.341389] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 959.008218] Key type lgssc registered [ 960.070309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 967.814818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 967.835459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.377592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 973.474708] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 979.945974] LustreError: 34970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.988682] LustreError: 34970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 982.325884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.640665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 987.700953] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 995.934122] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1002.364336] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1009.476324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1009.894387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.031428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1015.779516] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1023.141875] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1035.746416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1035.754296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.758242] LustreError: Skipped 1 previous similar message [ 1035.791578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1037.793720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1037.813609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1037.826134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.196437] Lustre: server umount lustre-OST0000 complete [ 1048.040736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.072085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1048.081801] Lustre: Skipped 2 previous similar messages [ 1051.616935] Lustre: server umount lustre-MDT0000 complete [ 1053.154472] LustreError: 34966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.187343] LustreError: 34966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1055.334825] LustreError: 34951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441457 with bad export cookie 17677705348939582320 [ 1055.336280] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1055.346078] LustreError: 34951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1055.620971] Lustre: server umount lustre-MDT0001 complete [ 1065.626409] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 1068.472194] Key type lgssc unregistered [ 1068.772606] LNet: 38507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.784731] LNetError: 38507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.824019] LNet: Removed LNI 192.168.206.149@tcp [ 1069.570825] Key type .llcrypt unregistered [ 1069.573906] Key type ._llcrypt unregistered [ 1094.128298] Key type ._llcrypt registered [ 1094.130485] Key type .llcrypt registered [ 1094.205809] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 1095.167055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1095.219048] alg: No test for adler32 (adler32-zlib) [ 1096.197963] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1096.412858] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 1098.080327] Key type lgssc registered [ 1099.646752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1108.709916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1108.723499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.126624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.008612] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1120.745025] LustreError: 39922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.798243] LustreError: 39922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1122.761945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.046646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1127.760661] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1135.241137] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1141.724654] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1149.068587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1149.418303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.577047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1156.034190] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1163.858727] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1175.937286] Lustre: server umount lustre-OST0000 complete [ 1191.393468] Lustre: 39234:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441577/real 1788441577] req@00000000f385944c x1875316832235520/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788441593 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1191.416409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1194.528786] Lustre: 39234:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441580/real 1788441580] req@00000000d19df544 x1875316832236544/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788441596 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1194.531121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1194.562529] Lustre: 39234:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1196.578846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.593030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.598288] Lustre: Skipped 1 previous similar message [ 1199.267685] Lustre: 39234:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441585/real 1788441585] req@000000008c304a8c x1875316832237568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788441601 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1199.587935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.596924] Lustre: Skipped 1 previous similar message [ 1201.641593] Lustre: server umount lustre-MDT0000 complete [ 1204.707659] LustreError: 39922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1204.723480] LustreError: 39922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1204.770035] Lustre: 39235:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441590/real 1788441590] req@00000000d6d33802 x1875316832238720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788441606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1205.295179] LustreError: 39901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441607 with bad export cookie 11331456293988762440 [ 1205.311107] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1211.954662] Lustre: server umount lustre-MDT0001 complete [ 1223.540499] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 1226.584605] Key type lgssc unregistered [ 1226.906182] LNet: 43635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.913856] LNetError: 43635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.928034] LNet: Removed LNI 192.168.206.149@tcp [ 1227.858922] Key type .llcrypt unregistered [ 1227.861922] Key type ._llcrypt unregistered [ 1253.308565] Key type ._llcrypt registered [ 1253.315360] Key type .llcrypt registered [ 1253.500907] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 1254.498489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.522936] alg: No test for adler32 (adler32-zlib) [ 1255.653346] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1255.856473] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 1257.528193] Key type lgssc registered [ 1258.738216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1266.931159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1266.960940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.344469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.937794] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1278.954308] LustreError: 45048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.977856] LustreError: 45048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1281.240330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.883364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1287.321534] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1297.331703] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1305.918615] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1312.921592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1313.321189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.383039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1319.180846] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1326.660754] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1333.219728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1333.241442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.263636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.289589] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.49@tcp (stopping) [ 1335.447379] Lustre: server umount lustre-MDT0000 complete [ 1336.804988] LustreError: 45048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.806448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.846297] Lustre: Skipped 1 previous similar message [ 1339.283271] LustreError: 45027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441741 with bad export cookie 8385675854335462814 [ 1339.295886] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1339.316913] LustreError: 45027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1339.379345] LustreError: 45043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.402236] LustreError: 45043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1339.433970] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.49@tcp (stopping) [ 1341.923767] LustreError: 46543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.928849] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.955298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1345.636554] Lustre: server umount lustre-MDT0001 complete [ 1376.225814] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1376.513884] Lustre: server umount lustre-OST0000 complete [ 1391.766666] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 1394.350675] Key type lgssc unregistered [ 1394.671787] LNet: 48909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.677915] LNetError: 48909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.692114] LNet: Removed LNI 192.168.206.149@tcp [ 1395.394212] Key type .llcrypt unregistered [ 1395.396474] Key type ._llcrypt unregistered [ 1417.769152] Key type ._llcrypt registered [ 1417.771479] Key type .llcrypt registered [ 1417.851996] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 1418.729534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1418.846975] alg: No test for adler32 (adler32-zlib) [ 1419.888783] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1420.129486] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 1421.792139] Key type lgssc registered [ 1422.613416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1428.793954] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1428.818580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1444.385219] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1449.504488] LustreError: 50298:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.149@tcp: failed processing log, type 4: rc = -110 [ 1476.192259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.664756] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1490.576570] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1656.910441] Lustre: server umount lustre-OST0000 complete [ 1673.007898] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 1675.425598] Key type lgssc unregistered [ 1675.674796] LNet: 51980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.685333] LNetError: 51980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.702963] LNet: Removed LNI 192.168.206.149@tcp [ 1676.525923] Key type .llcrypt unregistered [ 1676.532430] Key type ._llcrypt unregistered [ 1701.525809] Key type ._llcrypt registered [ 1701.534824] Key type .llcrypt registered [ 1701.636726] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 1702.470351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1702.522464] alg: No test for adler32 (adler32-zlib) [ 1703.679138] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1704.047456] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 1705.768265] Key type lgssc registered [ 1706.934286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1713.983945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1714.008068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.402087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1719.554264] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1725.930349] LustreError: 53396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.951580] LustreError: 53396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1728.060269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.563149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1732.983263] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1740.140583] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1745.819425] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1752.758042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1753.329905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.515395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1761.018691] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1768.358622] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1775.077436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1775.093647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.111661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1776.104758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1776.109416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1776.120374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.405862] Lustre: server umount lustre-OST0000 complete [ 1781.229283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.232115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1781.254941] Lustre: Skipped 1 previous similar message [ 1781.255345] Lustre: Skipped 1 previous similar message [ 1786.347905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.357024] Lustre: Skipped 2 previous similar messages [ 1786.894728] Lustre: server umount lustre-MDT0000 complete [ 1790.315726] LustreError: 53378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442192 with bad export cookie 13387030518870596123 [ 1790.318346] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1790.327239] LustreError: 53378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1790.682960] Lustre: server umount lustre-MDT0001 complete [ 1801.143518] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 1803.787178] Key type lgssc unregistered [ 1804.092961] LNet: 56930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.102298] LNetError: 56930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.123966] LNet: Removed LNI 192.168.206.149@tcp [ 1804.959257] Key type .llcrypt unregistered [ 1804.964843] Key type ._llcrypt unregistered [ 1825.934708] Key type ._llcrypt registered [ 1825.941352] Key type .llcrypt registered [ 1826.049766] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 1827.117650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1827.160499] alg: No test for adler32 (adler32-zlib) [ 1828.240725] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1828.533475] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 1830.184153] Key type lgssc registered [ 1831.152731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1838.370294] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1838.399735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1854.048403] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1859.168330] LustreError: 58308:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.149@tcp: failed processing log, type 4: rc = -110 [ 1885.856363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.138812] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1899.202960] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1905.330884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1905.765384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1905.853992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1909.544612] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1910.755330] LustreError: 58334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.778855] LustreError: 58334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1915.876734] LustreError: 58333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1917.959814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.303462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1922.681660] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1930.416849] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1935.522405] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1938.914982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1938.929839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.952071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1940.962335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.976108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1940.986184] Lustre: Skipped 1 previous similar message [ 1943.532516] Lustre: server umount lustre-OST0000 complete [ 1966.435268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.440465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1966.456099] Lustre: Skipped 1 previous similar message [ 1966.465314] Lustre: Skipped 1 previous similar message [ 1971.687388] LustreError: 59351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.704536] LustreError: 59351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1971.793333] Lustre: server umount lustre-MDT0000 complete [ 1975.450874] LustreError: 58314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442377 with bad export cookie 8215359740442904984 [ 1975.451553] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1975.460106] LustreError: 58314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1981.824212] Lustre: server umount lustre-MDT0001 complete [ 1992.988676] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 1995.916509] Key type lgssc unregistered [ 1996.219795] LNet: 62056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1996.230459] LNetError: 62056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1996.244887] LNet: Removed LNI 192.168.206.149@tcp [ 1997.054648] Key type .llcrypt unregistered [ 1997.056078] Key type ._llcrypt unregistered [ 2019.605916] Key type ._llcrypt registered [ 2019.610798] Key type .llcrypt registered [ 2019.690900] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 2020.755352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2020.815102] alg: No test for adler32 (adler32-zlib) [ 2021.892694] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2022.204126] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 2023.880178] Key type lgssc registered [ 2025.047500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2032.774527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2032.795590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.302541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.618817] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2044.901431] LustreError: 63467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.919563] LustreError: 63467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2047.801186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.339247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2053.028550] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2061.198817] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2066.759235] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2073.747245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2074.025066] LustreError: 65291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.153958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.184575] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2080.238536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2087.862738] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2105.827039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.834823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2105.853987] Lustre: Skipped 1 previous similar message [ 2107.620747] Lustre: server umount lustre-OST0000 complete [ 2115.041311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2115.049901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.073769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2115.084531] Lustre: Skipped 1 previous similar message [ 2116.070138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.086474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2116.089107] Lustre: Skipped 1 previous similar message [ 2117.789484] Lustre: server umount lustre-MDT0000 complete [ 2121.182561] LustreError: 63449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442523 with bad export cookie 18123094671296402673 [ 2121.195705] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2121.206372] LustreError: 63449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2121.566066] Lustre: server umount lustre-MDT0001 complete [ 2133.224950] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 2136.177343] Key type lgssc unregistered [ 2136.512378] LNet: 67031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.528856] LNetError: 67031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2136.550259] LNet: Removed LNI 192.168.206.149@tcp [ 2137.551693] Key type .llcrypt unregistered [ 2137.553798] Key type ._llcrypt unregistered [ 2163.145885] Key type ._llcrypt registered [ 2163.153214] Key type .llcrypt registered [ 2163.324680] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 2164.380187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2164.416897] alg: No test for adler32 (adler32-zlib) [ 2165.517527] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2165.751225] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 2167.456341] Key type lgssc registered [ 2168.550625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2177.055862] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:36:16 (1788442576) [ 2183.796458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2183.824404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.328535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.712334] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2195.942697] LustreError: 68744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.960807] LustreError: 68744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2197.804221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.172946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2202.144084] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2210.179776] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2217.071651] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2224.059905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2224.531032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.716540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2230.981039] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2238.808927] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2248.523098] Lustre: Failing over lustre-MDT0000 [ 2248.876979] Lustre: server umount lustre-MDT0000 complete [ 2249.186866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.194698] LustreError: 68740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.197080] Lustre: Skipped 2 previous similar messages [ 2249.223665] LustreError: 68740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2251.824545] LustreError: 68739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.317340] LustreError: 69983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.347671] LustreError: 69983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2259.430237] LustreError: 68744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.444403] LustreError: 68744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2260.322950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.471311] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.609440] Lustre: *** cfs_fail_loc=135, val=0*** [ 2260.704429] LustreError: 71831:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2260.722440] LustreError: 71831:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.149@tcp: cfg command failed: rc = -2 [ 2260.740213] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2260.740213] [ 2260.760672] LustreError: MGC192.168.206.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2260.779423] LustreError: 71819:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2260.797528] LustreError: 71819:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2260.810768] LustreError: 71819:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2260.827330] LustreError: 71819:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2260.918608] Lustre: server umount lustre-MDT0000 complete [ 2260.923794] LustreError: 71819:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2268.242914] LustreError: 68739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.285440] LustreError: 68739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2268.377875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2268.567512] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2268.582720] LustreError: 68736:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2268.906918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.021316] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2272.237972] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2273.232831] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2274.282153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2274.318453] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2274.342732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2284.517105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.521734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2284.531618] Lustre: Skipped 1 previous similar message [ 2284.543094] Lustre: Skipped 1 previous similar message [ 2286.711318] Lustre: server umount lustre-OST0000 complete [ 2294.766162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2294.766790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.770320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2294.781729] Lustre: Skipped 1 previous similar message [ 2296.622301] Lustre: server umount lustre-MDT0000 complete [ 2299.884550] LustreError: 68743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.919202] LustreError: 68743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2301.006779] LustreError: 68723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442702 with bad export cookie 13362579553811482067 [ 2301.014184] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2301.021820] LustreError: 68723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2301.497820] Lustre: server umount lustre-MDT0001 complete [ 2312.437476] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 2315.115949] Key type lgssc unregistered [ 2315.333600] LNet: 73613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.339271] LNetError: 73613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2315.353565] LNet: Removed LNI 192.168.206.149@tcp [ 2316.011381] Key type .llcrypt unregistered [ 2316.017596] Key type ._llcrypt unregistered [ 2341.049777] Key type ._llcrypt registered [ 2341.053453] Key type .llcrypt registered [ 2341.218402] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 2342.815255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.971357] alg: No test for adler32 (adler32-zlib) [ 2344.268891] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2344.510301] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 2346.225895] Key type lgssc registered [ 2347.449892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2355.928392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2355.945643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.399142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.354665] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2367.971165] LustreError: 75024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.005772] LustreError: 75024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2370.737296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.138091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2375.882822] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2383.541205] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2389.369070] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2396.801952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2397.077515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.234301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2403.542527] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2411.563464] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2421.176279] Lustre: Failing over lustre-MDT0000 [ 2421.489339] Lustre: server umount lustre-MDT0000 complete [ 2422.241143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2422.250285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.272352] LustreError: 76026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.352591] LustreError: 75021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.828890] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.425721] LustreError: 76025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.449199] LustreError: 76025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2431.420235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.630263] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2431.852967] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2431.989520] LustreError: 78111:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2432.001398] LustreError: 78111:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.149@tcp: cfg command failed: rc = -9 [ 2432.009893] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2432.009893] [ 2432.020836] LustreError: MGC192.168.206.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2432.037173] LustreError: 78099:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2432.048308] LustreError: 78099:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2432.067598] LustreError: 78099:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2432.078197] LustreError: 78099:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2432.104917] Lustre: server umount lustre-MDT0000 complete [ 2432.106669] LustreError: 78099:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2434.548742] LustreError: 75021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.569887] LustreError: 75021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2439.271424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.484660] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2439.505762] LustreError: 78459:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.206.149@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000045b31667) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2439.541448] LustreError: 75017:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2439.672466] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.49@tcp (not set up) [ 2439.835321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2439.899342] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2444.649141] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2444.798064] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2445.298477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2445.348472] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2445.400637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2455.527733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.529989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2455.547247] Lustre: Skipped 2 previous similar messages [ 2455.566548] Lustre: Skipped 1 previous similar message [ 2460.650360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2460.664267] Lustre: Skipped 1 previous similar message [ 2465.762087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2465.771832] Lustre: Skipped 1 previous similar message [ 2467.296763] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2467.525190] Lustre: server umount lustre-OST0000 complete [ 2476.004121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.005381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2476.022483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2477.943820] Lustre: server umount lustre-MDT0000 complete [ 2481.121731] LustreError: 75021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.144269] LustreError: 75021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2482.428920] LustreError: 75004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442884 with bad export cookie 15567007425683808778 [ 2482.445144] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2482.450223] LustreError: 75004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2482.936666] Lustre: server umount lustre-MDT0001 complete [ 2495.628957] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 2498.628779] Key type lgssc unregistered [ 2499.004396] LNet: 79898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2499.016567] LNetError: 79898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2499.042620] LNet: Removed LNI 192.168.206.149@tcp [ 2500.060784] Key type .llcrypt unregistered [ 2500.068571] Key type ._llcrypt unregistered [ 2513.809317] Key type ._llcrypt registered [ 2513.811412] Key type .llcrypt registered [ 2513.906869] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 2514.160302] Key type .llcrypt unregistered [ 2514.164094] Key type ._llcrypt unregistered [ 2540.052412] Key type ._llcrypt registered [ 2540.055452] Key type .llcrypt registered [ 2540.167121] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 2541.507748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2541.559837] alg: No test for adler32 (adler32-zlib) [ 2542.630293] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2543.018523] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 2544.744234] Key type lgssc registered [ 2545.697766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2553.536603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2553.550227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.128043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2559.390792] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2565.604768] LustreError: 81808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.621047] LustreError: 81808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2567.344692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.726331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2573.469628] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2582.256831] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2588.881701] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2596.345701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2596.662299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.773669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2602.620970] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2609.508892] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2623.460638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2623.463449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2623.475765] Lustre: Skipped 1 previous similar message [ 2623.476162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2628.587460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2628.609537] Lustre: Skipped 2 previous similar messages [ 2628.853205] Lustre: server umount lustre-OST0000 complete [ 2633.698643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.709070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2633.724263] Lustre: Skipped 1 previous similar message [ 2633.739580] Lustre: Skipped 1 previous similar message [ 2638.819366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2638.830694] Lustre: Skipped 1 previous similar message [ 2639.273985] Lustre: server umount lustre-MDT0000 complete [ 2643.317036] LustreError: 81790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443045 with bad export cookie 11606114527823258195 [ 2643.325475] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2643.340268] LustreError: 81790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2643.654795] Lustre: server umount lustre-MDT0001 complete [ 2655.092632] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 2658.356748] Key type lgssc unregistered [ 2658.675803] LNet: 85366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2658.683132] LNetError: 85366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2658.711286] LNet: Removed LNI 192.168.206.149@tcp [ 2659.637484] Key type .llcrypt unregistered [ 2659.642570] Key type ._llcrypt unregistered [ 2687.125680] Key type ._llcrypt registered [ 2687.127758] Key type .llcrypt registered [ 2687.237741] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 2688.266606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2688.356731] alg: No test for adler32 (adler32-zlib) [ 2689.516669] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2689.847938] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 2691.512244] Key type lgssc registered [ 2692.751649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2701.622995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2701.656700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.681278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2708.302674] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2714.089966] LustreError: 86780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.101867] LustreError: 86780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2718.711752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.090355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2724.367786] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2731.878968] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2737.776419] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2744.597384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2744.919928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.076995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2751.629332] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2758.810851] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2768.866126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2768.871022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.885167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2768.903611] Lustre: Skipped 1 previous similar message [ 2770.181893] Lustre: server umount lustre-OST0000 complete [ 2775.522562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2775.537052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.566629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.571416] Lustre: Skipped 1 previous similar message [ 2779.106491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2779.109717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2779.150910] Lustre: Skipped 1 previous similar message [ 2780.585729] Lustre: server umount lustre-MDT0000 complete [ 2784.226845] LustreError: 86780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.246542] LustreError: 86780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2784.427762] LustreError: 87518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443186 with bad export cookie 4606305125766174022 [ 2784.428914] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2784.437593] LustreError: 87518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2784.687046] Lustre: server umount lustre-MDT0001 complete [ 2795.813265] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 2798.495886] Key type lgssc unregistered [ 2798.782052] LNet: 90316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2798.796184] LNetError: 90316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2798.821541] LNet: Removed LNI 192.168.206.149@tcp [ 2799.732554] Key type .llcrypt unregistered [ 2799.736976] Key type ._llcrypt unregistered [ 2821.809842] Key type ._llcrypt registered [ 2821.812557] Key type .llcrypt registered [ 2821.904669] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 2822.961850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2822.978793] alg: No test for adler32 (adler32-zlib) [ 2824.108949] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2824.358696] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 2826.008201] Key type lgssc registered [ 2827.039702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2834.457936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2834.476560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.879935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.469601] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2845.162374] LustreError: 91729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2845.197343] LustreError: 91729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2848.652929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.942338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2853.114090] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2860.141524] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2865.257254] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2871.964112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2872.321123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.635324] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2880.518265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2887.711984] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2898.917556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2898.926532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.940549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2899.942801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2899.954289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2899.973303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2904.364897] Lustre: server umount lustre-OST0000 complete [ 2910.691263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2910.709056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2910.727297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2910.732822] Lustre: Skipped 2 previous similar messages [ 2914.337706] Lustre: server umount lustre-MDT0000 complete [ 2916.321090] LustreError: 92457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.336866] LustreError: 92457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2917.638824] LustreError: 91711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443319 with bad export cookie 17638840750667227873 [ 2917.655976] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2918.018219] Lustre: server umount lustre-MDT0001 complete [ 2928.321896] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 2931.173516] Key type lgssc unregistered [ 2931.488645] LNet: 95266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2931.499289] LNetError: 95266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2932.522719] LNet: Removed LNI 192.168.206.149@tcp [ 2933.603374] Key type .llcrypt unregistered [ 2933.605775] Key type ._llcrypt unregistered [ 2959.099480] Key type ._llcrypt registered [ 2959.102657] Key type .llcrypt registered [ 2959.216401] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 2960.502288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2960.527552] alg: No test for adler32 (adler32-zlib) [ 2961.603511] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2961.813950] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 2963.472634] Key type lgssc registered [ 2964.816765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2972.713608] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2972.732911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2988.321069] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2993.440301] LustreError: 96663:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.149@tcp: failed processing log, type 4: rc = -110 [ 3020.192608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3026.307815] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3033.700323] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3039.271504] Lustre: server umount lustre-OST0000 complete [ 3047.706458] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:50:47 (1788443447) [ 3053.773542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.208705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.121119] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3059.688264] LustreError: 98304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.708678] LustreError: 98304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3064.812499] LustreError: 98303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.487830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.887674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3071.333474] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3078.788390] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3084.180402] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3090.447430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3090.810145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3096.559850] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3100.024771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3103.289816] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3115.501022] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3115.515122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3115.537765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3118.562304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3118.567810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3118.575239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.007912] Lustre: server umount lustre-OST0000 complete [ 3123.700124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.713079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3123.723747] Lustre: Skipped 1 previous similar message [ 3128.289363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3128.308797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.340894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.036980] Lustre: server umount lustre-MDT0000 complete [ 3133.925328] LustreError: 101211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.958920] LustreError: 101211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3134.266455] LustreError: 98284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443536 with bad export cookie 3514930300621363728 [ 3134.281130] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.287379] LustreError: 98284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3134.565470] Lustre: server umount lustre-MDT0001 complete [ 3146.331770] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 3149.765863] Key type lgssc unregistered [ 3150.184642] LNet: 101996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3150.195290] LNetError: 101996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3150.216684] LNet: Removed LNI 192.168.206.149@tcp [ 3151.183696] Key type .llcrypt unregistered [ 3151.191095] Key type ._llcrypt unregistered [ 3197.115865] Key type ._llcrypt registered [ 3197.117757] Key type .llcrypt registered [ 3197.245582] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 3198.681942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3198.722988] alg: No test for adler32 (adler32-zlib) [ 3199.834537] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3200.200697] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 3201.928307] Key type lgssc registered [ 3202.924733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3211.226680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3211.245861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.592937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.732915] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3223.015074] LustreError: 104188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.033878] LustreError: 104188:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3227.817340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.130243] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3228.294614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3233.476917] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3241.588652] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3247.844373] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3254.387408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3254.707269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3257.787649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3260.469554] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3268.265303] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3283.435372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.438918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3283.462835] LustreError: Skipped 1 previous similar message [ 3283.468380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3288.553906] LustreError: 106009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.568608] LustreError: 106009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3288.675946] Lustre: server umount lustre-OST0000 complete [ 3294.690907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3294.709100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.724977] Lustre: Skipped 1 previous similar message [ 3294.738298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.746084] Lustre: Skipped 1 previous similar message [ 3298.792902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3298.796263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.829856] Lustre: server umount lustre-MDT0000 complete [ 3303.923308] LustreError: 104189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.961065] LustreError: 104189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3304.928888] LustreError: 104169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443706 with bad export cookie 13736617849539148847 [ 3304.945095] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3305.269652] Lustre: server umount lustre-MDT0001 complete [ 3316.335657] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 3319.227342] Key type lgssc unregistered [ 3319.573711] LNet: 107746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3319.595284] LNetError: 107746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3320.618117] LNet: Removed LNI 192.168.206.149@tcp [ 3321.476671] Key type .llcrypt unregistered [ 3321.479543] Key type ._llcrypt unregistered [ 3338.634210] Key type ._llcrypt registered [ 3338.640273] Key type .llcrypt registered [ 3338.728379] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 3339.755812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3339.801229] alg: No test for adler32 (adler32-zlib) [ 3340.843269] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3341.124866] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 3342.800971] Key type lgssc registered [ 3343.824679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3351.127566] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3351.147799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3366.688389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3371.808298] LustreError: 108988:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.149@tcp: failed processing log, type 4: rc = -110 [ 3398.502986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.341393] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3411.440856] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3417.855402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3418.138735] LustreError: MGC192.168.206.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3418.151841] LustreError: 110014:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3418.157781] LustreError: 110014:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3418.170191] LustreError: 110014:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3418.254155] Lustre: server umount lustre-MDT0000 complete [ 3418.257542] LustreError: 110014:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3436.112425] Lustre: server umount lustre-OST0000 complete [ 3443.945868] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 3451.882822] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 3486.522797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3486.742876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3486.775590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3486.857411] Lustre: lustre-MDT0000: new disk, initializing [ 3486.925657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.945928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3491.177400] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3501.721392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.851259] Lustre: 112805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3501.899881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3501.904334] Lustre: Skipped 1 previous similar message [ 3501.976603] Lustre: lustre-MDT0001: new disk, initializing [ 3502.076620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3502.107494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3502.121209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3506.150469] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3515.084833] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3520.504523] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3526.583961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3526.907759] Lustre: lustre-OST0000: new disk, initializing [ 3526.914926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3526.924348] Lustre: 114775:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3527.066904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3533.332709] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3534.383534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3534.393410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3534.541057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3542.451288] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3547.476246] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3547.705240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3552.356157] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3552.589193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3554.785635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3554.796329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3554.818190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3555.316945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3555.333930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3555.358309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3555.366901] Lustre: Skipped 1 previous similar message [ 3560.439649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3560.844859] Lustre: server umount lustre-OST0000 complete [ 3565.544105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3565.556343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3565.565716] Lustre: Skipped 2 previous similar messages [ 3570.659888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3570.667960] Lustre: Skipped 1 previous similar message [ 3579.360142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3579.554703] Lustre: server umount lustre-MDT0000 complete [ 3584.092557] LustreError: 112796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443986 with bad export cookie 12831253681281081673 [ 3584.098463] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3584.100607] LustreError: 112796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3584.552682] Lustre: server umount lustre-MDT0001 complete [ 3596.765481] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:59:55 (1788443995) [ 3601.559967] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3617.435234] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 3627.378514] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 3664.391803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3664.601394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3664.624759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3664.684235] Lustre: lustre-MDT0000: new disk, initializing [ 3664.746199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3664.768901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3669.373750] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3679.249456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3679.338528] Lustre: 119687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3679.417680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3679.424384] Lustre: Skipped 1 previous similar message [ 3679.545342] Lustre: lustre-MDT0001: new disk, initializing [ 3679.639024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3679.680393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3679.694409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3683.437910] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3692.467608] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3698.299656] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3707.226580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3707.495596] Lustre: lustre-OST0000: new disk, initializing [ 3707.500295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3707.508714] Lustre: 121652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3707.607742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.441795] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3717.171853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3717.190863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3717.256709] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3723.945246] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3728.339544] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3728.500146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3732.809097] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3733.053334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3738.081377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3738.092937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3738.095171] LustreError: Skipped 1 previous similar message [ 3738.095708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3738.095712] Lustre: Skipped 2 previous similar messages [ 3738.132462] Lustre: Skipped 2 previous similar messages [ 3749.346682] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3749.505225] Lustre: server umount lustre-OST0000 complete [ 3753.446445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.561374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3758.567124] Lustre: Skipped 8 previous similar messages [ 3767.264216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3767.598043] Lustre: server umount lustre-MDT0000 complete [ 3768.801077] LustreError: 119698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.813885] LustreError: 119698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3772.081357] LustreError: 119679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444174 with bad export cookie 12831253681281082555 [ 3772.082409] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3772.114352] LustreError: 119679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3772.475556] Lustre: server umount lustre-MDT0001 complete [ 3781.187548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3781.501476] LustreError: 123871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3781.575240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3785.462284] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3786.724207] LustreError: 123872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.854529] LustreError: 123871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3793.793908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3798.172355] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3805.811388] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3811.694679] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3819.209942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3826.278964] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3835.888043] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3846.245847] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3852.773065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3852.778989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3852.788759] Lustre: Skipped 1 previous similar message [ 3852.795021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3852.799337] Lustre: Skipped 2 previous similar messages [ 3856.609137] Lustre: server umount lustre-OST0000 complete [ 3860.961997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3860.970546] LustreError: Skipped 1 previous similar message [ 3874.272132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3874.508293] Lustre: server umount lustre-MDT0000 complete [ 3874.785789] LustreError: 124599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3874.811435] LustreError: 124599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3878.649568] LustreError: 125717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444280 with bad export cookie 12831253681281083437 [ 3878.655674] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3878.666381] LustreError: 125717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3879.139356] Lustre: server umount lustre-MDT0001 complete [ 3891.831018] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 3894.068842] Key type lgssc unregistered [ 3894.296295] LNet: 127509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3894.305904] LNetError: 127509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3894.326326] LNet: Removed LNI 192.168.206.149@tcp [ 3895.022078] Key type .llcrypt unregistered [ 3895.024295] Key type ._llcrypt unregistered [ 3911.150801] Key type ._llcrypt registered [ 3911.153150] Key type .llcrypt registered [ 3911.252700] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 3929.326782] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 3931.038289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3931.381541] alg: No test for adler32 (adler32-zlib) [ 3932.754154] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3933.329553] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 3935.144162] Key type lgssc registered [ 3936.328728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3971.847913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3971.859660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3973.097362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3973.137493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3973.236635] Lustre: lustre-MDT0000: new disk, initializing [ 3973.346854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.367721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3978.309618] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3989.263551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3989.402859] Lustre: 130659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3989.465984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3989.468615] Lustre: Skipped 1 previous similar message [ 3989.609849] Lustre: lustre-MDT0001: new disk, initializing [ 3989.739421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3989.802161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3989.815571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3994.325067] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4003.260180] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4009.048711] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4017.877290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4018.234705] Lustre: lustre-OST0000: new disk, initializing [ 4018.241705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4018.255286] Lustre: 132629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4018.437919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.856535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4023.871366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4024.008866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4025.061375] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4034.378737] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4038.830792] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4039.051507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4043.535424] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4043.770694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4049.889275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4049.894071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.925044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4049.937024] Lustre: Skipped 1 previous similar message [ 4051.868168] Lustre: server umount lustre-OST0000 complete [ 4056.551695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4056.559833] LustreError: Skipped 1 previous similar message [ 4056.569544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4056.579062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4056.588079] Lustre: Skipped 1 previous similar message [ 4060.138509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4060.150486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.888495] Lustre: server umount lustre-MDT0000 complete [ 4065.865429] LustreError: 130652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444467 with bad export cookie 11155025279272389869 [ 4065.878350] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4065.882480] LustreError: 130652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4066.494291] Lustre: server umount lustre-MDT0001 complete [ 4075.303831] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:07:54 (1788444474) [ 4081.841315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4082.389544] LustreError: 135042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4082.405096] LustreError: 135042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4082.429729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4087.272838] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4087.777220] LustreError: 135043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4092.921768] LustreError: 135042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4094.877203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4095.129487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4098.585499] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4106.851654] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4112.774159] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4115.938271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4115.942210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.954114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4115.965109] Lustre: Skipped 1 previous similar message [ 4121.062488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4121.070155] Lustre: Skipped 3 previous similar messages [ 4121.409494] Lustre: server umount lustre-MDT0000 complete [ 4125.260544] LustreError: 135022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444527 with bad export cookie 11155025279272390751 [ 4125.264418] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4125.268227] LustreError: 135022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4125.497591] Lustre: server umount lustre-MDT0001 complete [ 4133.507707] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:08:52 (1788444532) [ 4139.082948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4154.720997] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4159.841766] LustreError: 137459:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.149@tcp: failed processing log, type 4: rc = -110 [ 4185.568309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4192.408839] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4200.342675] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4201.998920] Lustre: server umount lustre-OST0000 complete [ 4210.128061] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:10:09 (1788444609) [ 4216.585557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4217.204264] LustreError: 138904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4217.332473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4221.716217] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4222.447953] LustreError: 138905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.944606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4235.785576] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4242.839430] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4248.186724] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4253.737937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4254.071937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.079172] Lustre: Skipped 1 previous similar message [ 4259.634911] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4266.624233] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4277.730150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4277.735516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4277.741825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4277.762883] Lustre: Skipped 3 previous similar messages [ 4277.766533] Lustre: Skipped 1 previous similar message [ 4282.574897] Lustre: server umount lustre-MDT0000 complete [ 4282.850216] LustreError: 139635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4282.876870] LustreError: 139635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4285.967444] LustreError: 138886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444687 with bad export cookie 11155025279272391409 [ 4285.984360] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4285.996445] LustreError: 138886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4286.321657] Lustre: server umount lustre-MDT0001 complete [ 4300.271881] Lustre: server umount lustre-OST0000 complete [ 4309.649446] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:11:49 (1788444709) [ 4315.650549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.962438] LustreError: 142560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.955927] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4329.080118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4329.458229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4329.465882] Lustre: Skipped 1 previous similar message [ 4334.334886] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4342.283972] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4349.105415] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4355.114869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4360.572957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4361.411261] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4369.225286] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4373.471780] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4373.711732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4378.358902] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4378.603654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4381.156195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4381.163276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.176863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4381.182121] Lustre: Skipped 1 previous similar message [ 4386.612962] Lustre: server umount lustre-OST0000 complete [ 4396.699259] Lustre: server umount lustre-MDT0000 complete [ 4400.506342] LustreError: 142542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444802 with bad export cookie 11155025279272392487 [ 4400.514731] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4400.528096] LustreError: 142542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4400.935060] Lustre: server umount lustre-MDT0001 complete [ 4411.292421] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:13:30 (1788444810) [ 4418.196233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4433.824575] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4438.944735] LustreError: 146718:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.149@tcp: failed processing log, type 4: rc = -110 [ 4464.672263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.684305] Lustre: Skipped 1 previous similar message [ 4471.029183] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4478.181484] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4484.771878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4485.110439] LustreError: 146743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.123191] LustreError: 146743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4485.244870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4489.659600] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4499.103625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4499.432171] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4499.442416] Lustre: Skipped 7 previous similar messages [ 4504.747067] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4513.239682] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4518.892252] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4523.752586] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4523.990734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4529.292844] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4529.549850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4535.265925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4535.273259] LustreError: Skipped 1 previous similar message [ 4535.284137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.305016] Lustre: Skipped 3 previous similar messages [ 4537.650071] Lustre: server umount lustre-MDT0000 complete [ 4541.487431] LustreError: 146726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444943 with bad export cookie 11155025279272393509 [ 4541.495789] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4541.504342] LustreError: 146726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4541.821727] Lustre: server umount lustre-MDT0001 complete [ 4555.725065] Lustre: server umount lustre-OST0000 complete [ 4563.500853] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:16:03 (1788444963) [ 4568.801456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.416850] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4589.538041] LustreError: 150914:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.149@tcp: failed processing log, type 4: rc = -110 [ 4623.656837] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4632.232170] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4639.698925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.031876] LustreError: 150939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4640.047375] LustreError: 150939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4640.145199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4644.928101] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4653.383541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4658.022462] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4664.950654] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4670.451645] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4676.557951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4676.726580] Lustre: lustre-OST0001: new disk, initializing [ 4676.730946] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4676.743789] Lustre: 153787:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4681.608204] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4682.280152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4682.291786] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4682.436935] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4690.059246] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4694.698975] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4694.939217] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4700.343030] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4700.613296] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4703.208687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4703.225531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4703.241258] Lustre: Skipped 3 previous similar messages [ 4703.255020] Lustre: Skipped 4 previous similar messages [ 4717.024949] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4717.113038] Lustre: server umount lustre-OST0000 complete [ 4723.685933] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4723.690107] LustreError: Skipped 1 previous similar message [ 4726.985780] Lustre: server umount lustre-OST0001 complete [ 4745.191227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4745.435289] Lustre: server umount lustre-MDT0000 complete [ 4749.053019] LustreError: 150920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445150 with bad export cookie 11155025279272394461 [ 4749.062492] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4749.064021] LustreError: 150920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4749.484516] Lustre: server umount lustre-MDT0001 complete [ 4770.465776] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:19:29 (1788445169) [ 4772.741782] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4774.660852] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4776.459717] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:19:35 (1788445175) [ 4785.120573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4785.384410] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4785.412909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4785.640463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4785.646103] Lustre: Skipped 6 previous similar messages [ 4789.608836] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4797.657844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4797.753952] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4797.795785] Lustre: 157348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4802.124122] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4808.597331] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4814.559674] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4821.407649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4821.721468] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4827.345017] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4828.650214] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4840.225596] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4845.811643] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4846.102375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4852.647374] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4852.882222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4857.825464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4857.838122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4857.877466] Lustre: Skipped 4 previous similar messages [ 4857.898747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4857.906825] Lustre: Skipped 12 previous similar messages [ 4860.890886] Lustre: server umount lustre-OST0000 complete [ 4881.695406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4884.009270] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4884.034270] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4888.707770] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4896.027396] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4898.408355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4898.424747] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4900.949645] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4901.140407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4905.054974] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4905.249637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4910.275032] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9547a02d8800.ost_server_uuid 50 [ 4911.365557] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9547a02d8800.ost_server_uuid in FULL state after 0 sec [ 4911.575939] LustreError: 157354:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.49@tcp) returned error from blocking AST (req@000000007d032dce x1875319807278080 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000a00fbbe0/0x9ace9c63bf88a837 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.49@tcp remote: 0xe30d443df30a538 expref: 5 pid: 158179 timeout: 5011 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4911.612616] LustreError: lustre-MDT0000: A client on nid 192.168.206.49@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4911.620091] LustreError: 157344:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.49@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000a00fbbe0/0x9ace9c63bf88a837 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.49@tcp remote: 0xe30d443df30a538 expref: 6 pid: 158179 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4921.242939] Lustre: server umount lustre-OST0000 complete [ 4930.337841] Lustre: server umount lustre-MDT0000 complete [ 4933.017530] LustreError: 157340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445334 with bad export cookie 11155025279272395644 [ 4933.019232] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4933.023260] LustreError: 157340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4939.297817] Lustre: server umount lustre-MDT0001 complete [ 4947.994919] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 4950.172842] Key type lgssc unregistered [ 4950.444396] LNet: 163463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4950.456399] LNetError: 163463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4950.477136] LNet: Removed LNI 192.168.206.149@tcp [ 4951.192996] Key type .llcrypt unregistered [ 4951.195106] Key type ._llcrypt unregistered [ 4974.182201] Key type ._llcrypt registered [ 4974.186993] Key type .llcrypt registered [ 4974.326949] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 4975.034359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4975.157933] alg: No test for adler32 (adler32-zlib) [ 4976.143796] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4976.318529] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 4978.004388] Key type lgssc registered [ 4978.967746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4986.474454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4986.495228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4987.962174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4992.340352] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 4997.093192] LustreError: 164880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4997.111338] LustreError: 164880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5000.841391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5001.169152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5006.194628] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5014.790432] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5022.114785] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5031.795793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5032.439293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5033.424823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5041.821365] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5049.327916] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5056.553079] Lustre: Failing over lustre-MDT0000 [ 5056.867786] Lustre: server umount lustre-MDT0000 complete [ 5056.995687] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.005037] LustreError: 164875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5057.025036] LustreError: 164875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5062.126866] LustreError: 164880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5062.161368] LustreError: 164880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5067.233547] LustreError: 164876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5067.261086] LustreError: 164876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5069.635345] LustreError: 164859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.49@tcp arrived at 1788445471 with bad export cookie 5580202709812034524 [ 5069.663350] LustreError: 164859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5072.362200] LustreError: 164879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5072.387226] LustreError: 164879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5078.258497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5078.404836] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5078.662780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5078.749325] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5083.805947] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5084.141861] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5084.158261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5086.328240] Lustre: lustre-MDT0000: Denying connection for new client 7c8b7c27-22ac-4652-9616-1209d4da40ad (at 192.168.206.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5097.958968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5097.965810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.977016] Lustre: Skipped 2 previous similar messages [ 5097.981776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5099.491096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5099.495943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5104.615088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5104.629642] Lustre: Skipped 2 previous similar messages [ 5109.216169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5109.383323] Lustre: server umount lustre-OST0000 complete [ 5113.612258] LustreError: 168611:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5113.619393] Lustre: 167959:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5113.629656] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5113.653267] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5113.664823] LustreError: 167959:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f3a8dc9f x1875320900731136/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5113.695326] LustreError: 167959:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5113.712259] LustreError: 167959:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5113.727899] Lustre: lustre-MDT0000: Recovery over after 0:29, of 2 clients 0 recovered and 2 were evicted. [ 5114.121374] Lustre: server umount lustre-MDT0000 complete [ 5114.850644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.873503] LustreError: 164875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5114.898686] LustreError: 164875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5118.165981] LustreError: 164858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445520 with bad export cookie 5580202709812034622 [ 5118.179839] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5118.188353] LustreError: 164858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5120.511948] Lustre: server umount lustre-MDT0001 complete [ 5132.649110] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 5135.933849] Key type lgssc unregistered [ 5136.268839] LNet: 169366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5136.279384] LNetError: 169366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5136.302416] LNet: Removed LNI 192.168.206.149@tcp [ 5137.427167] Key type .llcrypt unregistered [ 5137.429072] Key type ._llcrypt unregistered [ 5165.278041] Key type ._llcrypt registered [ 5165.280838] Key type .llcrypt registered [ 5165.456704] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 5166.756610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5166.832066] alg: No test for adler32 (adler32-zlib) [ 5167.868969] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5168.169411] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 5169.936623] Key type lgssc registered [ 5171.367597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5180.278992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5180.295753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5181.819796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5187.110956] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5191.142328] LustreError: 170780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.176345] LustreError: 170780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5196.265389] LustreError: 170781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5197.144433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5197.781983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5203.286991] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5212.693071] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5220.074813] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5227.905407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5228.354205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5235.409040] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5244.475663] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5251.908245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5262.319432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5262.333405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5262.336769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5262.377602] Lustre: Skipped 1 previous similar message [ 5267.427936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5267.450221] Lustre: Skipped 1 previous similar message [ 5268.036971] Lustre: server umount lustre-OST0000 complete [ 5272.546865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5272.570668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5272.578506] Lustre: Skipped 2 previous similar messages [ 5277.665390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5277.671755] Lustre: Skipped 1 previous similar message [ 5282.785920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5282.799980] Lustre: Skipped 1 previous similar message [ 5286.368136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5286.724806] Lustre: server umount lustre-MDT0000 complete [ 5287.918844] LustreError: 170777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5287.945641] LustreError: 170777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5290.856395] LustreError: 173280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445692 with bad export cookie 12016471912241032546 [ 5290.870825] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5290.873446] LustreError: 173280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5291.312057] Lustre: server umount lustre-MDT0001 complete [ 5303.728441] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 5306.597550] Key type lgssc unregistered [ 5306.862698] LNet: 174337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5306.867470] LNetError: 174337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5306.889944] LNet: Removed LNI 192.168.206.149@tcp [ 5307.674430] Key type .llcrypt unregistered [ 5307.676732] Key type ._llcrypt unregistered [ 5339.432537] Key type ._llcrypt registered [ 5339.437159] Key type .llcrypt registered [ 5339.585182] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 5341.742759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5341.778477] alg: No test for adler32 (adler32-zlib) [ 5342.883853] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5343.220711] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 5344.928192] Key type lgssc registered [ 5346.284313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5354.396620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5354.432462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5355.939623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5359.517692] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5365.222718] LustreError: 176044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.263907] LustreError: 176044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5367.496759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.935957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5373.634449] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5381.772087] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5388.944704] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5396.159066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5396.384256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5400.504285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5402.263150] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5409.380437] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5423.082280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5423.087497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5423.091041] LustreError: Skipped 1 previous similar message [ 5423.094778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5423.104094] Lustre: Skipped 1 previous similar message [ 5423.115503] Lustre: Skipped 1 previous similar message [ 5427.649214] Lustre: server umount lustre-OST0000 complete [ 5433.313418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.327267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5433.331777] Lustre: Skipped 1 previous similar message [ 5437.653704] Lustre: server umount lustre-MDT0000 complete [ 5438.433634] LustreError: 176044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5440.890288] LustreError: 176024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445842 with bad export cookie 15736695620432449840 [ 5440.907504] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5441.174602] Lustre: server umount lustre-MDT0001 complete [ 5450.247644] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 5452.740037] Key type lgssc unregistered [ 5452.976766] LNet: 179601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5452.984098] LNetError: 179601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5454.058346] LNet: Removed LNI 192.168.206.149@tcp [ 5454.640804] Key type .llcrypt unregistered [ 5454.643468] Key type ._llcrypt unregistered [ 5477.049191] Key type ._llcrypt registered [ 5477.051595] Key type .llcrypt registered [ 5477.149026] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 5477.993572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5478.016635] alg: No test for adler32 (adler32-zlib) [ 5479.006364] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5479.234815] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 5480.912955] Key type lgssc registered [ 5481.715321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5493.372615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5493.421429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5494.993584] Lustre: *** cfs_fail_loc=135, val=0*** [ 5495.077138] LustreError: 181124:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5495.090343] LustreError: 181124:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.149@tcp: cfg command failed: rc = -2 [ 5495.098837] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5495.098837] [ 5495.107311] LustreError: MGC192.168.206.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5495.116422] LustreError: 181086:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5495.125734] LustreError: 181086:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5495.135430] LustreError: 181086:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5495.140365] LustreError: 181086:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5495.180234] Lustre: server umount lustre-MDT0000 complete [ 5495.190027] LustreError: 181086:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5502.492102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.087486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5507.748723] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5512.173782] LustreError: 181498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5512.192761] LustreError: 181498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5515.955820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5516.266753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5519.685409] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5528.136851] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5533.768243] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5535.591301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5535.602308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5535.610783] Lustre: Skipped 1 previous similar message [ 5535.625754] Lustre: Skipped 1 previous similar message [ 5540.837693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5540.847048] Lustre: Skipped 1 previous similar message [ 5541.783120] Lustre: server umount lustre-MDT0000 complete [ 5541.862549] LustreError: 181498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5541.901995] LustreError: 181498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5545.707730] LustreError: 181479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445947 with bad export cookie 4967321564074315783 [ 5545.710322] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5545.718956] LustreError: 181479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5546.287770] Lustre: server umount lustre-MDT0001 complete [ 5559.433782] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 5562.510501] Key type lgssc unregistered [ 5562.752847] LNet: 183823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5562.767277] LNetError: 183823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5563.821734] LNet: Removed LNI 192.168.206.149@tcp [ 5564.410903] Key type .llcrypt unregistered [ 5564.415161] Key type ._llcrypt unregistered [ 5588.519402] Key type ._llcrypt registered [ 5588.523766] Key type .llcrypt registered [ 5588.673948] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 5589.143773] Key type .llcrypt unregistered [ 5589.147600] Key type ._llcrypt unregistered [ 5606.573133] Key type ._llcrypt registered [ 5606.575517] Key type .llcrypt registered [ 5606.665524] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 5607.840232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5607.862956] alg: No test for adler32 (adler32-zlib) [ 5608.946614] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5609.226856] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 5610.945792] Key type lgssc registered [ 5612.048933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5620.850924] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5620.903112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5636.516220] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5641.632350] LustreError: 185960:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.149@tcp: failed processing log, type 4: rc = -110 [ 5668.320492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5674.271424] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5683.082615] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5689.965237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5690.390738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5690.462854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5694.569866] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5695.463653] LustreError: 185986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.487632] LustreError: 185986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5700.584109] LustreError: 185985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5702.629746] LustreError: 186999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5702.644844] LustreError: 186999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5702.885964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.180313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5707.270772] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5713.010935] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5717.747341] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5720.284478] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5728.742931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5728.743874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.749133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5728.757077] Lustre: Skipped 1 previous similar message [ 5733.877243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5733.884707] Lustre: Skipped 2 previous similar messages [ 5734.122545] Lustre: server umount lustre-OST0000 complete [ 5738.987488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5738.990780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5739.009243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5739.025107] Lustre: Skipped 1 previous similar message [ 5743.923300] Lustre: server umount lustre-MDT0000 complete [ 5744.102722] LustreError: 187003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5744.132297] LustreError: 187003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5747.409791] LustreError: 185966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446149 with bad export cookie 14622453151068088696 [ 5747.414762] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5747.419430] LustreError: 185966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5747.902429] Lustre: server umount lustre-MDT0001 complete [ 5758.660169] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 5761.894669] Key type lgssc unregistered [ 5762.247587] LNet: 189879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5762.263294] LNetError: 189879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5763.304386] LNet: Removed LNI 192.168.206.149@tcp [ 5764.313155] Key type .llcrypt unregistered [ 5764.316424] Key type ._llcrypt unregistered [ 5787.478272] Key type ._llcrypt registered [ 5787.481435] Key type .llcrypt registered [ 5787.588993] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 5788.441886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5788.539148] alg: No test for adler32 (adler32-zlib) [ 5789.547490] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5789.824849] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 5791.489315] Key type lgssc registered [ 5792.528772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5800.136891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5800.159970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.612757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5806.315519] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5811.172539] LustreError: 191291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.184341] LustreError: 191291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5815.930810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.229978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5821.579770] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5831.038313] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5837.766943] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5845.352143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5845.726752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.804270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5851.991564] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5858.549216] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5862.878962] Lustre: Failing over lustre-MDT0000 [ 5863.068301] Lustre: server umount lustre-MDT0000 complete [ 5865.488216] LustreError: 191285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5866.469451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5866.485281] Lustre: Skipped 2 previous similar messages [ 5866.489463] LustreError: 191290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5866.500769] LustreError: 191290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5870.578206] LustreError: 192454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5875.711399] LustreError: 191285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5875.743657] LustreError: 191285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5881.413293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5881.563449] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5881.805662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5881.886760] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5881.908957] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5886.764285] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5886.973390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5887.007336] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 5887.067403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5891.305836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5900.871917] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5915.355720] Lustre: server umount lustre-OST0000 complete [ 5922.786307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5922.789946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5922.822691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5922.831268] Lustre: Skipped 1 previous similar message [ 5927.906733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5927.914736] Lustre: Skipped 1 previous similar message [ 5929.015020] Lustre: server umount lustre-MDT0000 complete [ 5932.432692] LustreError: 191271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446334 with bad export cookie 6402872820548661827 [ 5932.442357] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5932.445165] LustreError: 191271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5942.880722] Lustre: server umount lustre-MDT0001 complete [ 5952.717180] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 5955.547703] Key type lgssc unregistered [ 5955.832706] LNet: 196251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5955.840545] LNetError: 196251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5956.903740] LNet: Removed LNI 192.168.206.149@tcp [ 5957.748843] Key type .llcrypt unregistered [ 5957.751968] Key type ._llcrypt unregistered [ 5980.057285] Key type ._llcrypt registered [ 5980.061153] Key type .llcrypt registered [ 5980.178709] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 5981.336378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5981.435868] alg: No test for adler32 (adler32-zlib) [ 5982.540959] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5982.874444] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 5984.632170] Key type lgssc registered [ 5985.679745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5992.861643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5992.897791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5994.426263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.283553] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6004.716483] LustreError: 197656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6004.744440] LustreError: 197656:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6009.069921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6009.479157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6013.532286] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6022.025603] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6028.622620] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6035.214231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6035.743594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6039.860237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6042.205236] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6048.861783] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6052.742849] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6057.353395] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6070.293216] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6079.971330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6079.977712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.979909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6079.990791] Lustre: Skipped 1 previous similar message [ 6080.005208] Lustre: Skipped 1 previous similar message [ 6082.312901] Lustre: server umount lustre-OST0000 complete [ 6086.121670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6086.132939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6086.153342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6090.212716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6090.216930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.904417] Lustre: server umount lustre-MDT0000 complete [ 6095.198258] LustreError: 200444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446497 with bad export cookie 15287520871888677306 [ 6095.202268] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6095.207734] LustreError: 200444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6095.471235] Lustre: server umount lustre-MDT0001 complete [ 6104.596903] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 6107.448422] Key type lgssc unregistered [ 6107.724806] LNet: 201501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6107.733047] LNetError: 201501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6107.752178] LNet: Removed LNI 192.168.206.149@tcp [ 6108.747278] Key type .llcrypt unregistered [ 6108.762633] Key type ._llcrypt unregistered [ 6127.700725] Key type ._llcrypt registered [ 6127.705848] Key type .llcrypt registered [ 6127.805577] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 6128.766174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6128.797540] alg: No test for adler32 (adler32-zlib) [ 6129.887071] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6130.133481] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 6131.777876] Key type lgssc registered [ 6132.711383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6139.122414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6139.133866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6140.601200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6144.457401] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6150.125254] LustreError: 202767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6150.142432] LustreError: 202767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6152.885517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6153.294603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6157.355429] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6164.977680] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6170.537192] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6177.216839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6177.596246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6183.648875] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6185.771256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6191.233887] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6197.840932] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6211.108527] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6220.587239] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6223.779843] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6235.117245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6235.123940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6235.135175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6237.159075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6237.161960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6237.165628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6239.840529] Lustre: server umount lustre-OST0000 complete [ 6245.345692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6245.353415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6245.369493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6245.374798] Lustre: Skipped 1 previous similar message [ 6247.393186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6247.396310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6249.364427] Lustre: server umount lustre-MDT0000 complete [ 6252.377133] LustreError: 203501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446654 with bad export cookie 11407557373981363579 [ 6252.386095] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6252.389669] LustreError: 203501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6252.571141] Lustre: server umount lustre-MDT0001 complete [ 6262.877697] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 6265.776615] Key type lgssc unregistered [ 6266.026959] LNet: 207742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6266.032799] LNetError: 207742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6266.049465] LNet: Removed LNI 192.168.206.149@tcp [ 6266.883331] Key type .llcrypt unregistered [ 6266.887326] Key type ._llcrypt unregistered [ 6290.101548] Key type ._llcrypt registered [ 6290.108565] Key type .llcrypt registered [ 6290.224641] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 6291.548962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6291.567054] alg: No test for adler32 (adler32-zlib) [ 6292.611889] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6292.856673] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 6294.544875] Key type lgssc registered [ 6295.654807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6303.068341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6303.096619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.654065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6309.600861] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6314.980726] LustreError: 209158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6315.000835] LustreError: 209158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6318.959589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6319.411180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6324.310505] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6331.964323] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6337.924918] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6344.722602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6345.090661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6349.174059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6351.365143] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6357.935949] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6374.884864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6374.886045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.907746] LustreError: Skipped 1 previous similar message [ 6374.909341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6374.934525] Lustre: Skipped 1 previous similar message [ 6374.966990] Lustre: Skipped 1 previous similar message [ 6377.419513] Lustre: server umount lustre-OST0000 complete [ 6381.026128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6381.031880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.050362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6385.123810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6385.129407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6390.241612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6390.253561] Lustre: Skipped 2 previous similar messages [ 6394.338478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6394.345612] Lustre: Skipped 1 previous similar message [ 6395.360134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6395.495333] Lustre: server umount lustre-MDT0000 complete [ 6398.771375] LustreError: 209139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446800 with bad export cookie 1107535567610772305 [ 6398.774236] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6398.785613] LustreError: 209139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6399.177128] Lustre: server umount lustre-MDT0001 complete [ 6408.838703] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 6411.719349] Key type lgssc unregistered [ 6412.033551] LNet: 212845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6412.043093] LNetError: 212845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6412.067347] LNet: Removed LNI 192.168.206.149@tcp [ 6412.769981] Key type .llcrypt unregistered [ 6412.772134] Key type ._llcrypt unregistered [ 6429.459706] Key type ._llcrypt registered [ 6429.465378] Key type .llcrypt registered [ 6429.603400] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 6431.111370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6431.149229] alg: No test for adler32 (adler32-zlib) [ 6432.232226] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6432.709222] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 6434.432266] Key type lgssc registered [ 6435.733369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6443.798484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6443.822378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6445.205546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6448.785806] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6455.783959] LustreError: 214092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6455.806505] LustreError: 214092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6457.022123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6457.494668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6461.850107] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6469.278619] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6474.309444] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6480.157087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6480.515353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6486.051759] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6490.752504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6493.188937] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6508.520374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6508.536733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6508.563258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6511.585585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6511.596445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6511.598441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6511.610907] Lustre: Skipped 1 previous similar message [ 6513.174727] Lustre: server umount lustre-OST0000 complete [ 6516.706990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.718689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6516.722239] Lustre: Skipped 1 previous similar message [ 6516.751332] Lustre: Skipped 1 previous similar message [ 6521.826645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6522.583663] Lustre: server umount lustre-MDT0000 complete [ 6525.987670] LustreError: 216734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446927 with bad export cookie 17873285032220119023 [ 6526.001950] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6526.009389] LustreError: 216734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6526.387337] Lustre: server umount lustre-MDT0001 complete [ 6537.724739] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 6541.179615] Key type lgssc unregistered [ 6541.434859] LNet: 217777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6541.456260] LNetError: 217777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6541.494894] LNet: Removed LNI 192.168.206.149@tcp [ 6542.409154] Key type .llcrypt unregistered [ 6542.410886] Key type ._llcrypt unregistered [ 6560.430227] Key type ._llcrypt registered [ 6560.431794] Key type .llcrypt registered [ 6560.512281] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 6561.167478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6561.240680] alg: No test for adler32 (adler32-zlib) [ 6562.303871] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6562.519441] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 6564.216347] Key type lgssc registered [ 6565.296693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6572.607739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6572.632767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6574.035680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6577.956474] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6583.275890] LustreError: 219043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.301201] LustreError: 219043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6585.757349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.068565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6589.902709] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6597.607461] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6604.138399] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6610.759615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6610.933867] LustreError: 220864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6611.129674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6617.682369] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6618.610967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6624.610988] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6632.788458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6632.869631] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6633.119326] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6639.439373] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6648.035728] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6649.723897] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6661.370271] Lustre: Permanently deactivating lustre-OST0001 [ 6661.379599] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6668.415739] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6675.070032] Lustre: Permanently reactivating lustre-OST0001 [ 6675.093405] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6675.098888] Lustre: Skipped 2 previous similar messages [ 6676.203630] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.216165] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6676.236176] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6676.266322] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6676.316635] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6689.846583] Lustre: server umount lustre-OST0001 complete [ 6691.808996] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6691.814947] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6691.819314] LustreError: Skipped 1 previous similar message [ 6691.824972] LustreError: 220865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6691.837024] Lustre: Skipped 2 previous similar messages [ 6695.393837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6695.407205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6695.416299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6696.931658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6696.935198] LustreError: 220864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6696.967141] LustreError: 220864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6699.706259] Lustre: server umount lustre-OST0000 complete [ 6704.102104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6704.113664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6704.129817] Lustre: Skipped 1 previous similar message [ 6704.137836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6704.142361] Lustre: Skipped 1 previous similar message [ 6707.171950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6707.177477] Lustre: Skipped 1 previous similar message [ 6709.284362] Lustre: server umount lustre-MDT0000 complete [ 6712.290084] LustreError: 219038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6712.300539] LustreError: 219038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6712.511886] LustreError: 219024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447114 with bad export cookie 16917511161001615969 [ 6712.513638] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6712.520041] LustreError: 219024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6712.890557] Lustre: server umount lustre-MDT0001 complete [ 6723.035678] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 6725.775366] Key type lgssc unregistered [ 6726.109535] LNet: 224115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6726.123571] LNetError: 224115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6726.138234] LNet: Removed LNI 192.168.206.149@tcp [ 6727.219514] Key type .llcrypt unregistered [ 6727.224250] Key type ._llcrypt unregistered [ 6763.231807] Key type ._llcrypt registered [ 6763.234063] Key type .llcrypt registered [ 6763.374623] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 6764.164159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6764.238323] alg: No test for adler32 (adler32-zlib) [ 6765.313289] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6765.553533] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 6767.209044] Key type lgssc registered [ 6768.233467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6775.329954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6775.347509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6776.594591] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6776.624871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6776.862674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6781.472952] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6788.661376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6788.784068] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6788.807894] Lustre: 226298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6789.028734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6792.976910] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6801.143170] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6806.669481] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6812.350984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6812.505720] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6812.712862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6818.657115] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 6818.999219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6825.623753] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6829.602256] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6836.404196] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6844.960857] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6853.695592] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6870.872370] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6870.878140] Lustre: Skipped 1 previous similar message [ 6889.916229] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6889.923283] Lustre: Skipped 2 previous similar messages [ 6928.573790] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6928.581412] Lustre: Skipped 4 previous similar messages [ 6992.569695] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6992.582303] Lustre: Skipped 7 previous similar messages [ 7003.114893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7003.128305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7003.140387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7003.624608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7003.643670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7003.677094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7005.849861] Lustre: server umount lustre-OST0000 complete [ 7013.858361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.885129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7013.890390] Lustre: Skipped 1 previous similar message [ 7015.844882] Lustre: server umount lustre-MDT0000 complete [ 7019.849974] LustreError: 226290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447421 with bad export cookie 10505835396409617431 [ 7019.852317] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7019.879116] LustreError: 226290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7020.342400] Lustre: server umount lustre-MDT0001 complete [ 7033.256686] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 7035.438298] Key type lgssc unregistered [ 7035.641599] LNet: 231090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7035.651248] LNetError: 231090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7035.669969] LNet: Removed LNI 192.168.206.149@tcp [ 7036.325500] Key type .llcrypt unregistered [ 7036.328871] Key type ._llcrypt unregistered [ 7056.307328] Key type ._llcrypt registered [ 7056.310317] Key type .llcrypt registered [ 7056.395281] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 7057.351903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7057.366053] alg: No test for adler32 (adler32-zlib) [ 7058.413759] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7058.633967] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 7060.296147] Key type lgssc registered [ 7061.239189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7067.620122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7067.634520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.025862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7072.508839] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 7078.375186] LustreError: 232483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7078.391419] LustreError: 232483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7080.342352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7080.734088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7085.225526] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 7093.058424] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7098.936657] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7105.539759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7105.968802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7111.015473] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 7113.137545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7117.188072] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7138.786071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7138.796861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.807116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7140.836660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7140.840546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7140.840878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7140.880553] Lustre: Skipped 1 previous similar message [ 7143.387567] Lustre: server umount lustre-OST0000 complete [ 7147.501325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7147.517844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7147.540326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7151.075700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7151.078140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7156.207927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7156.215379] Lustre: Skipped 2 previous similar messages [ 7161.313709] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7161.588387] Lustre: server umount lustre-MDT0000 complete [ 7165.411947] LustreError: 232479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7165.432340] LustreError: 232479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7165.508144] LustreError: 235062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447567 with bad export cookie 12445763204106851925 [ 7165.515632] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7165.523522] LustreError: 235062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7165.814425] Lustre: server umount lustre-MDT0001 complete [ 7177.347173] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 7180.106224] Key type lgssc unregistered [ 7180.365245] LNet: 236180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7180.373959] LNetError: 236180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7180.406227] LNet: Removed LNI 192.168.206.149@tcp [ 7181.312581] Key type .llcrypt unregistered [ 7181.315061] Key type ._llcrypt unregistered [ 7200.379988] Key type ._llcrypt registered [ 7200.382502] Key type .llcrypt registered [ 7200.482488] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 7201.374316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7201.437132] alg: No test for adler32 (adler32-zlib) [ 7202.586517] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7202.971768] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 7204.656310] Key type lgssc registered [ 7205.801052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7478.163787] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 7480.893396] Key type lgssc unregistered [ 7481.224716] LNet: 238200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7481.231726] LNetError: 238200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7481.264622] LNet: Removed LNI 192.168.206.149@tcp [ 7482.155511] Key type .llcrypt unregistered [ 7482.157415] Key type ._llcrypt unregistered [ 7509.178770] Key type ._llcrypt registered [ 7509.182123] Key type .llcrypt registered [ 7509.302453] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 7524.645194] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 7526.065687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7526.088321] alg: No test for adler32 (adler32-zlib) [ 7527.204877] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7527.480480] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 7529.136226] Key type lgssc registered [ 7530.312883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7561.326848] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7294 sec ========== 11:06:00 (1788447960) [ 7563.332716] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:06:02 (1788447962) === [ 7566.571333] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:06:06 (1788447966) === [ 7607.603576] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 7610.491841] Key type lgssc unregistered [ 7610.827189] LNet: 243439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7610.835882] LNetError: 243439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7610.884855] LNet: Removed LNI 192.168.206.149@tcp [ 7612.232559] Key type .llcrypt unregistered [ 7612.236889] Key type ._llcrypt unregistered