[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 610802748 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001021] APIC: Switch to symmetric I/O mode setup [ 0.002378] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.011046] LSM: Security Framework initializing [ 0.012060] Yama: becoming mindful. [ 0.013038] SELinux: Initializing. [ 0.014075] *** VALIDATE selinux *** [ 0.022713] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026692] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030016] *** VALIDATE tmpfs *** [ 0.031458] *** VALIDATE proc *** [ 0.033216] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036194] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043144] debug: unmapping init [mem 0xffffffffaae59000-0xffffffffaae60fff] [ 0.046285] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047675] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048024] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055275] rcu: Hierarchical SRCU implementation. [ 0.057598] smp: Bringing up secondary CPUs ... [ 0.058591] x86: Booting SMP configuration: [ 0.059032] .... node #0, CPUs: #1 #2 #3 [ 0.066022] smp: Brought up 1 node, 4 CPUs [ 0.068013] smpboot: Max logical packages: 1 [ 0.069020] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.135070] node 0 deferred pages initialised in 64ms [ 0.139126] devtmpfs: initialized [ 0.140231] x86/mm: Memory block size: 128MB [ 0.146806] gcov: version magic: 0x41383552 [ 0.151450] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156672] pinctrl core: initialized pinctrl subsystem [ 0.158289] [ 0.158910] ************************************************************* [ 0.160015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162018] ** ** [ 0.164015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166015] ** ** [ 0.169017] ** This means that this kernel is built to expose internal ** [ 0.171016] ** IOMMU data structures, which may compromise security on ** [ 0.173021] ** your system. ** [ 0.175019] ** ** [ 0.180016] ** If you see this message and you are not debugging the ** [ 0.182012] ** kernel, report this immediately to your vendor! ** [ 0.185012] ** ** [ 0.187015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189024] ************************************************************* [ 0.191865] NET: Registered protocol family 16 [ 0.193426] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203042] cpuidle: using governor menu [ 0.204999] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208667] PCI: Using configuration type 1 for base access [ 0.210133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.223133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228416] cryptd: max_cpu_qlen set to 1000 [ 0.230263] ACPI: Added _OSI(Module Device) [ 0.232014] ACPI: Added _OSI(Processor Device) [ 0.234024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242695] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.250379] ACPI: Interpreter enabled [ 0.252067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.254015] ACPI: Using IOAPIC for interrupt routing [ 0.256095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260824] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.275093] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.279046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.281018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.284075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.289598] acpiphp: Slot [2] registered [ 0.290210] acpiphp: Slot [5] registered [ 0.292138] acpiphp: Slot [6] registered [ 0.293162] acpiphp: Slot [7] registered [ 0.294131] acpiphp: Slot [8] registered [ 0.296116] acpiphp: Slot [9] registered [ 0.297159] acpiphp: Slot [10] registered [ 0.299199] acpiphp: Slot [3] registered [ 0.300094] acpiphp: Slot [4] registered [ 0.302138] acpiphp: Slot [11] registered [ 0.303158] acpiphp: Slot [12] registered [ 0.304095] acpiphp: Slot [13] registered [ 0.306162] acpiphp: Slot [14] registered [ 0.307092] acpiphp: Slot [15] registered [ 0.309110] acpiphp: Slot [16] registered [ 0.310087] acpiphp: Slot [17] registered [ 0.311122] acpiphp: Slot [18] registered [ 0.313083] acpiphp: Slot [19] registered [ 0.315091] acpiphp: Slot [20] registered [ 0.316085] acpiphp: Slot [21] registered [ 0.318088] acpiphp: Slot [22] registered [ 0.319080] acpiphp: Slot [23] registered [ 0.320107] acpiphp: Slot [24] registered [ 0.321111] acpiphp: Slot [25] registered [ 0.323122] acpiphp: Slot [26] registered [ 0.325189] acpiphp: Slot [27] registered [ 0.326099] acpiphp: Slot [28] registered [ 0.327134] acpiphp: Slot [29] registered [ 0.328097] acpiphp: Slot [30] registered [ 0.329097] acpiphp: Slot [31] registered [ 0.331059] PCI host bridge to bus 0000:00 [ 0.332021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.341022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344060] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347253] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350020] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353382] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.365747] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.372866] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.376031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.381018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.383369] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.386953] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.391046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.393762] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.400014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.419030] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.427019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.437157] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.451033] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.461035] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.487024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.508258] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.527016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.544022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.575018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.594483] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.607017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.616017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.645020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.664023] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.673016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.681017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.709016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.721632] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.736018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.748017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.778018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.793424] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.804018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.818022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.843019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.855744] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.858419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.860525] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.864427] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.867410] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.874035] iommu: Default domain type: Passthrough [ 0.875352] SCSI subsystem initialized [ 0.876000] ACPI: bus type USB registered [ 0.876137] usbcore: registered new interface driver usbfs [ 0.879110] usbcore: registered new interface driver hub [ 0.881106] usbcore: registered new device driver usb [ 0.883138] pps_core: LinuxPPS API ver. 1 registered [ 0.885014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.889061] PTP clock support registered [ 0.892161] EDAC MC: Ver: 3.0.0 [ 0.895435] PCI: Using ACPI for IRQ routing [ 0.898181] NetLabel: Initializing [ 0.900014] NetLabel: domain hash size = 128 [ 0.902012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.904090] NetLabel: unlabeled traffic allowed by default [ 0.907045] vgaarb: loaded [ 0.908331] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.911015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.916422] clocksource: Switched to clocksource kvm-clock [ 1.039821] VFS: Disk quotas dquot_6.6.0 [ 1.041955] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.045650] *** VALIDATE ramfs *** [ 1.047464] *** VALIDATE hugetlbfs *** [ 1.049712] pnp: PnP ACPI init [ 1.053335] pnp: PnP ACPI: found 6 devices [ 1.072903] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.077342] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.080232] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.083085] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.086239] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.089399] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.093070] NET: Registered protocol family 2 [ 1.096501] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.103823] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.109107] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.116622] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.121868] TCP: Hash tables configured (established 65536 bind 65536) [ 1.126642] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.131430] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.136139] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.140887] NET: Registered protocol family 1 [ 1.144766] RPC: Registered named UNIX socket transport module. [ 1.147550] RPC: Registered udp transport module. [ 1.150609] RPC: Registered tcp transport module. [ 1.152949] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.156370] NET: Registered protocol family 44 [ 1.158501] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.161503] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.164238] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.167648] PCI: CLS 0 bytes, default 64 [ 1.169948] Unpacking initramfs... [ 2.924741] debug: unmapping init [mem 0xffff9abffcc54000-0xffff9abffffbffff] [ 2.933322] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.936699] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.941738] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.536756] Initialise system trusted keyrings [ 3.538418] Key type blacklist registered [ 3.540936] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.551781] zbud: loaded [ 3.555348] *** VALIDATE nfs *** [ 3.556367] *** VALIDATE nfs4 *** [ 3.558160] pstore: using deflate compression [ 3.563031] Platform Keyring initialized [ 3.698651] NET: Registered protocol family 38 [ 3.701533] Key type asymmetric registered [ 3.704781] Asymmetric key parser 'x509' registered [ 3.707814] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.714301] io scheduler mq-deadline registered [ 3.716484] io scheduler kyber registered [ 3.719761] io scheduler bfq registered [ 3.725457] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.729669] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.733781] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.738446] ACPI: Power Button [PWRF] [ 3.745403] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.753883] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.787073] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.801781] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.836172] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.873516] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.909254] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.916979] Non-volatile memory driver v1.3 [ 3.920275] Linux agpgart interface v0.103 [ 3.959480] virtio_blk virtio1: [vda] 145800 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.962710] vda: detected capacity change from 0 to 74649600 [ 3.982115] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.986525] vdb: detected capacity change from 0 to 1073741824 [ 4.012688] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.017796] vdc: detected capacity change from 0 to 2621440000 [ 4.040497] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.046479] vdd: detected capacity change from 0 to 2621440000 [ 4.074632] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.080654] vde: detected capacity change from 0 to 4294967296 [ 4.101173] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.103981] vdf: detected capacity change from 0 to 4294967296 [ 4.113767] libphy: Fixed MDIO Bus: probed [ 4.128518] usbcore: registered new interface driver usbserial_generic [ 4.131440] usbserial: USB Serial support registered for generic [ 4.134429] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.140931] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.143275] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.147205] mousedev: PS/2 mouse device common for all mice [ 4.151493] rtc_cmos 00:05: RTC can wake from S4 [ 4.156753] rtc_cmos 00:05: registered as rtc0 [ 4.158369] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.162341] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.162400] intel_pstate: CPU model not supported [ 4.165470] hid: raw HID events driver (C) Jiri Kosina [ 4.174163] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.176791] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.178554] usbcore: registered new interface driver usbhid [ 4.178559] usbhid: USB HID core driver [ 4.178694] drop_monitor: Initializing network drop monitor service [ 4.178828] Initializing XFRM netlink socket [ 4.180604] NET: Registered protocol family 10 [ 4.205547] Segment Routing with IPv6 [ 4.207119] NET: Registered protocol family 17 [ 4.209632] mpls_gso: MPLS GSO support [ 4.216712] RAS: Correctable Errors collector initialized. [ 4.218754] AVX version of gcm_enc/dec engaged. [ 4.220835] AES CTR mode by8 optimization enabled [ 4.308607] sched_clock: Marking stable (4308526357, 0)->(5286549240, -978022883) [ 4.316985] registered taskstats version 1 [ 4.322656] Loading compiled-in X.509 certificates [ 4.324885] zswap: loaded using pool lzo/zbud [ 4.368774] Key type big_key registered [ 4.390698] Key type encrypted registered [ 4.392269] ima: No TPM chip found, activating TPM-bypass! [ 4.396627] ima: Allocated hash algorithm: sha1 [ 4.400027] ima: No architecture policies found [ 4.404403] evm: Initialising EVM extended attributes: [ 4.410669] evm: security.selinux [ 4.415344] evm: security.ima [ 4.416794] evm: security.capability [ 4.420378] evm: HMAC attrs: 0x1 [ 4.425361] rtc_cmos 00:05: setting system clock to 2026-07-26 14:38:18 UTC (1785076698) [ 4.439372] debug: unmapping init [mem 0xffffffffabe03000-0xffffffffabffffff] [ 4.444124] debug: unmapping init [mem 0xffffffffaab82000-0xffffffffaae58fff] [ 4.453078] Write protecting the kernel read-only data: 28672k [ 4.455939] debug: unmapping init [mem 0xffffffffa9203000-0xffffffffa93fffff] [ 4.459104] debug: unmapping init [mem 0xffffffffa9b14000-0xffffffffa9bfffff] [ 4.498712] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.510440] systemd[1]: Detected virtualization kvm. [ 4.512701] systemd[1]: Detected architecture x86-64. [ 4.514991] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.543417] systemd[1]: No hostname configured. [ 4.546604] systemd[1]: Set hostname to . [ 4.551589] random: systemd: uninitialized urandom read (16 bytes read) [ 4.554708] systemd[1]: Initializing machine ID from random generator. [ 4.649110] random: ln: uninitialized urandom read (6 bytes read) [ 4.789945] random: systemd: uninitialized urandom read (16 bytes read) [ 4.794104] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.806268] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 4.822470] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.751848] device-mapper: uevent: version 1.0.3 [ 5.753904] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.639326] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 6.684721] random: fast init done [ 6.990185] scsi host0: ata_piix [ 7.255172] scsi host1: ata_piix [ 7.259349] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.264925] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.929790] random: crng init done [ 11.931801] random: 7 urandom warning(s) missed due to ratelimiting [ 12.818290] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.995471] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.798636] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.189613] SELinux: Disabled at runtime. [ 17.266720] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.276828] systemd[1]: Detected virtualization kvm. [ 17.279147] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.092271] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.097607] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.111464] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.119377] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.126047] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.140444] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.156739] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ 18.361520] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 19.167941] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.993236] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.119360] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.463829] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.516355] EDAC sbridge: Ver: 1.1.2 [ 23.459274] Key type dns_resolver registered [ 23.819971] NFS: Registering the id_resolver key type [ 23.822773] Key type id_resolver registered [ 23.827926] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg445-server login: [ 76.790800] hrtimer: interrupt took 5726813 ns [ 80.904879] libcfs: loading out-of-tree module taints kernel. [ 80.984252] Key type ._llcrypt registered [ 80.985717] Key type .llcrypt registered [ 81.145535] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 103.678826] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 105.266306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.308534] alg: No test for adler32 (adler32-zlib) [ 106.952487] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 107.875103] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 109.650632] Key type lgssc registered [ 111.714426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 131.262264] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 181.102869] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 196.767796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 196.816869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.179899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 198.238150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 198.355461] Lustre: lustre-MDT0000: new disk, initializing [ 198.419530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.447375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.147146] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.989889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.120916] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.163469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.168675] Lustre: Skipped 1 previous similar message [ 220.309829] Lustre: lustre-MDT0001: new disk, initializing [ 220.422986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.483623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.497112] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 226.439928] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.638647] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 244.901061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.216773] Lustre: lustre-OST0000: new disk, initializing [ 245.222539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.233229] Lustre: 8453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.334873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.968000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 251.980713] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 252.073702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 253.902492] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.947680] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 272.082785] Lustre: lustre-OST0001: new disk, initializing [ 272.088315] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 272.095373] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 272.197296] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 279.637389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 279.661578] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 279.731493] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 280.081215] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.527850] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 300.963246] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 309.110525] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing check_logdir /tmp/testlogs/ [ 315.236586] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing yml_node [ 321.800681] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 325.113784] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 328.409428] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 330.294981] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Jul 26 10:43:41 EDT 2026 [ 350.067862] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 351.542337] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 369.120670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 369.131286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.145063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 372.193412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 372.194809] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.683047] Lustre: server umount lustre-MDT0000 complete [ 378.897620] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785077072 with bad export cookie 6621255568998473370 [ 378.898964] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 378.914138] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 382.433316] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 382.436615] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.480455] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 382.529688] Lustre: Skipped 2 previous similar messages [ 382.540595] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 382.550428] Lustre: Skipped 4 previous similar messages [ 383.456421] LustreError: 8925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.476829] LustreError: 8925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 385.682967] Lustre: server umount lustre-MDT0001 complete [ 390.742645] Lustre: server umount lustre-OST0000 complete [ 395.915867] Lustre: server umount lustre-OST0001 complete [ 404.505230] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 454.148823] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 502.762590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.034469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 503.079700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 503.204786] Lustre: lustre-MDT0000: new disk, initializing [ 503.316931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 503.336299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 508.603542] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 521.735497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.905458] Lustre: 15099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 522.035986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 522.047138] Lustre: Skipped 1 previous similar message [ 522.165571] Lustre: lustre-MDT0001: new disk, initializing [ 522.284120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 522.341392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 522.356453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 528.097251] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 541.068779] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 547.926152] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 555.759187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 556.066636] Lustre: lustre-OST0000: new disk, initializing [ 556.077406] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 556.095621] Lustre: 17061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 556.221201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 557.298641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 557.312620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 557.477538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 564.098312] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 574.554987] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 579.879228] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 580.128639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 585.964700] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 586.314059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 593.394312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 593.396583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.417019] LustreError: Skipped 1 previous similar message [ 593.421032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 593.438826] Lustre: Skipped 2 previous similar messages [ 593.456776] Lustre: Skipped 3 previous similar messages [ 594.967463] Lustre: server umount lustre-OST0000 complete [ 603.625256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.637437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.647134] Lustre: Skipped 1 previous similar message [ 603.664554] Lustre: Skipped 1 previous similar message [ 605.780885] Lustre: server umount lustre-MDT0000 complete [ 608.737403] LustreError: 15966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.779198] LustreError: 15966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 610.075228] LustreError: 15092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785077304 with bad export cookie 6621255568998474903 [ 610.082213] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.082913] LustreError: 15092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 610.496399] Lustre: server umount lustre-MDT0001 complete [ 614.418545] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 10:48:26 (1785077306) [ 622.237693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.918918] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.054413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.194784] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.224543] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 628.836914] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 637.906657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.483846] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 652.820130] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 659.390562] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 667.250416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 675.296521] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 685.352388] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 694.240615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 694.248334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.264353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 698.337504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 699.826650] Lustre: server umount lustre-OST0000 complete [ 704.997769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 711.060773] Lustre: server umount lustre-MDT0000 complete [ 713.696290] LustreError: 19320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.707775] LustreError: 19320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 715.347878] LustreError: 19301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785077409 with bad export cookie 6621255568998475778 [ 715.348441] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 715.359967] LustreError: 19301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 715.686396] Lustre: server umount lustre-MDT0001 complete [ 728.863899] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 733.178457] Key type lgssc unregistered [ 733.558602] LNet: 22842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.584270] LNetError: 22842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.639296] LNet: Removed LNI 192.168.204.145@tcp [ 734.873146] Key type .llcrypt unregistered [ 734.875104] Key type ._llcrypt unregistered [ 765.125332] Key type ._llcrypt registered [ 765.129290] Key type .llcrypt registered [ 765.336458] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 766.824953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 767.013847] alg: No test for adler32 (adler32-zlib) [ 768.149212] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 768.386514] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 770.103160] Key type lgssc registered [ 771.651293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 780.348565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 780.376202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 781.931952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 787.405694] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 791.523524] LustreError: 24264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 791.544022] LustreError: 24264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 796.654599] LustreError: 24265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 797.549584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.991533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 803.107288] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 811.597034] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 818.049515] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 825.405730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 825.833142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 826.910115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 832.803368] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 841.501281] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 860.318515] Lustre: server umount lustre-OST0000 complete [ 864.736747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 864.741960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.752248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 867.812359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 867.821845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.928600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 872.944814] Lustre: Skipped 2 previous similar messages [ 874.192085] Lustre: server umount lustre-MDT0000 complete [ 878.048744] LustreError: 24264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.048810] Lustre: 23577:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785077556/real 1785077556] req@00000000c2208832 x1871789384826496/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785077572 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 878.111636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.246437] LustreError: 24246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785077572 with bad export cookie 5919601531267550220 [ 878.259507] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 878.261895] LustreError: 24246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 878.700159] Lustre: server umount lustre-MDT0001 complete [ 890.778232] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 894.642191] Key type lgssc unregistered [ 894.939151] LNet: 28221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.955860] LNetError: 28221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.982345] LNet: Removed LNI 192.168.204.145@tcp [ 895.960961] Key type .llcrypt unregistered [ 895.966867] Key type ._llcrypt unregistered [ 919.589494] Key type ._llcrypt registered [ 919.592829] Key type .llcrypt registered [ 919.720155] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 921.062765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 921.203174] alg: No test for adler32 (adler32-zlib) [ 922.338727] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 922.665240] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 924.327178] Key type lgssc registered [ 925.545904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 934.878285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 934.913280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 936.488318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 941.995723] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 946.661184] LustreError: 29630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.688598] LustreError: 29630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 951.790777] LustreError: 29631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.469156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.874382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 957.802938] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 965.924516] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 971.933902] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 987.193386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 987.570339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 994.730840] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1003.576077] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1006.006814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1018.337499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1018.347712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.367588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.410796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.412151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.430724] Lustre: Skipped 1 previous similar message [ 1022.935277] Lustre: server umount lustre-OST0000 complete [ 1029.600581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1029.619546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.648562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1031.663980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.666937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1034.705694] Lustre: server umount lustre-MDT0000 complete [ 1036.774776] LustreError: 29625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1036.816209] LustreError: 29625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1039.913084] LustreError: 29613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785077733 with bad export cookie 3670503804426866842 [ 1039.919995] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1039.944084] LustreError: 29613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1040.350387] Lustre: server umount lustre-MDT0001 complete [ 1054.783458] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1057.943745] Key type lgssc unregistered [ 1058.258869] LNet: 33512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.275610] LNetError: 33512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.308534] LNet: Removed LNI 192.168.204.145@tcp [ 1059.410102] Key type .llcrypt unregistered [ 1059.414591] Key type ._llcrypt unregistered [ 1085.095844] Key type ._llcrypt registered [ 1085.099209] Key type .llcrypt registered [ 1085.205602] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1086.126979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1086.250864] alg: No test for adler32 (adler32-zlib) [ 1087.337160] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1087.582387] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1089.271169] Key type lgssc registered [ 1090.814930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1099.267527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1099.286454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.998728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.917653] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1110.504967] LustreError: 34917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1110.544803] LustreError: 34917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1115.622457] LustreError: 34918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.801852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.127372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1120.826558] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1129.812430] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1136.944584] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1145.411557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1145.881233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1150.975054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1154.101939] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1163.143592] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1176.544801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1176.556814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.566322] LustreError: Skipped 1 previous similar message [ 1176.581710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1176.609528] Lustre: Skipped 1 previous similar message [ 1181.284071] Lustre: server umount lustre-OST0000 complete [ 1186.784722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1186.831136] Lustre: Skipped 1 previous similar message [ 1186.848027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1186.862031] Lustre: Skipped 1 previous similar message [ 1191.919486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.929316] Lustre: Skipped 1 previous similar message [ 1192.904567] Lustre: server umount lustre-MDT0000 complete [ 1196.003031] LustreError: 37487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.045170] LustreError: 37487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1197.825551] LustreError: 34898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785077891 with bad export cookie 13672494306439496358 [ 1197.841522] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1198.383555] Lustre: server umount lustre-MDT0001 complete [ 1214.976556] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1218.594920] Key type lgssc unregistered [ 1218.989855] LNet: 38436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.003094] LNetError: 38436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.070247] LNet: Removed LNI 192.168.204.145@tcp [ 1221.164807] Key type .llcrypt unregistered [ 1221.166664] Key type ._llcrypt unregistered [ 1253.024821] Key type ._llcrypt registered [ 1253.030423] Key type .llcrypt registered [ 1253.193390] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1254.581113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.619373] alg: No test for adler32 (adler32-zlib) [ 1255.722116] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1255.992193] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1257.712849] Key type lgssc registered [ 1259.414248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1268.884280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1268.898441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.288540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.215514] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1280.484208] LustreError: 39847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1280.515028] LustreError: 39847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1285.601682] LustreError: 39848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1285.603622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.168285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1291.361728] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1299.609711] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1305.627388] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1313.819734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1314.399886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1315.448563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1322.771227] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1332.323339] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1339.562644] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1341.411835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1341.421464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.455852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1342.446483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1342.452454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1342.465273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1342.476034] Lustre: Skipped 1 previous similar message [ 1344.651994] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1344.876823] Lustre: server umount lustre-OST0000 complete [ 1364.350259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.360217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1370.781065] Lustre: server umount lustre-MDT0000 complete [ 1372.642327] LustreError: 39847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.668753] LustreError: 39847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1375.046648] LustreError: 41683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078069 with bad export cookie 7324905826476988500 [ 1375.053202] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1375.059483] LustreError: 41683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1381.672592] Lustre: server umount lustre-MDT0001 complete [ 1395.410590] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1398.385138] Key type lgssc unregistered [ 1398.726817] LNet: 43552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.732301] LNetError: 43552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1398.801058] LNet: Removed LNI 192.168.204.145@tcp [ 1399.891385] Key type .llcrypt unregistered [ 1399.895472] Key type ._llcrypt unregistered [ 1426.670796] Key type ._llcrypt registered [ 1426.678847] Key type .llcrypt registered [ 1426.805371] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1427.964144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1427.997796] alg: No test for adler32 (adler32-zlib) [ 1429.110596] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1429.468192] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1431.167150] Key type lgssc registered [ 1432.246336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1440.442152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1440.467505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.099913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1446.668899] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1452.512669] LustreError: 44959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.542412] LustreError: 44959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1456.295206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.786624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1461.723231] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1469.726601] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1477.581994] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1486.091088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1486.506445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.554955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1495.061425] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1503.627923] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1511.604214] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1512.425502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.430291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1512.436716] Lustre: Skipped 1 previous similar message [ 1512.446819] Lustre: Skipped 2 previous similar messages [ 1516.679166] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1517.538612] LustreError: 45960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.740113] Lustre: server umount lustre-MDT0000 complete [ 1521.549807] LustreError: 44939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078215 with bad export cookie 11054123259906424106 [ 1521.555939] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1521.556387] LustreError: 44939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1521.805493] LustreError: 44953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1521.817871] LustreError: 44953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1521.841173] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 1522.656488] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.668357] Lustre: Skipped 1 previous similar message [ 1526.688606] LustreError: 47059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.695475] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1526.710664] LustreError: 47059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1526.729834] Lustre: Skipped 1 previous similar message [ 1528.176593] Lustre: server umount lustre-MDT0001 complete [ 1550.930667] Lustre: server umount lustre-OST0000 complete [ 1574.347952] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1578.397673] Key type lgssc unregistered [ 1578.767540] LNet: 48803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.778491] LNetError: 48803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.815532] LNet: Removed LNI 192.168.204.145@tcp [ 1581.061375] Key type .llcrypt unregistered [ 1581.066419] Key type ._llcrypt unregistered [ 1612.179140] Key type ._llcrypt registered [ 1612.180918] Key type .llcrypt registered [ 1612.366923] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1613.562849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1613.610826] alg: No test for adler32 (adler32-zlib) [ 1614.734920] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1614.973218] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1616.671427] Key type lgssc registered [ 1618.008333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1627.759188] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1627.778596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1643.423457] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1648.543592] LustreError: 50186:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 1675.231983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.556883] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1691.215431] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1859.731537] Lustre: server umount lustre-OST0000 complete [ 1879.175930] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1882.857788] Key type lgssc unregistered [ 1883.354565] LNet: 51837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.359733] LNetError: 51837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.371713] LNet: Removed LNI 192.168.204.145@tcp [ 1884.513550] Key type .llcrypt unregistered [ 1884.521832] Key type ._llcrypt unregistered [ 1915.889695] Key type ._llcrypt registered [ 1915.892875] Key type .llcrypt registered [ 1916.058152] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1917.546534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1917.657682] alg: No test for adler32 (adler32-zlib) [ 1918.843481] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1919.092324] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1920.791208] Key type lgssc registered [ 1922.262851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1931.108347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1931.136626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.761100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.889160] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1941.985828] LustreError: 53247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.001374] LustreError: 53247:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1947.118157] LustreError: 53246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.835171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.184233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1953.441983] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1962.424069] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1970.638915] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1978.527699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1979.052775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.881891] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1996.937555] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2002.531642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2002.543081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.920501] Lustre: server umount lustre-OST0000 complete [ 2014.704397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2014.712217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.728247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.735759] Lustre: Skipped 1 previous similar message [ 2018.282909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.283582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2018.314978] Lustre: Skipped 1 previous similar message [ 2020.058070] Lustre: server umount lustre-MDT0000 complete [ 2023.393127] LustreError: 53242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2023.402500] LustreError: 53242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2023.988373] LustreError: 55084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078718 with bad export cookie 11534284946084883726 [ 2023.996183] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2024.005955] LustreError: 55084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2024.456365] Lustre: server umount lustre-MDT0001 complete [ 2036.923504] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2039.957379] Key type lgssc unregistered [ 2040.337092] LNet: 56766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2040.347552] LNetError: 56766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2040.365497] LNet: Removed LNI 192.168.204.145@tcp [ 2041.385662] Key type .llcrypt unregistered [ 2041.389760] Key type ._llcrypt unregistered [ 2065.152238] Key type ._llcrypt registered [ 2065.156645] Key type .llcrypt registered [ 2065.304981] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2066.606868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2066.760395] alg: No test for adler32 (adler32-zlib) [ 2067.852982] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2068.297849] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2070.023494] Key type lgssc registered [ 2071.614140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2081.096969] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2081.121246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2096.799493] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2101.919333] LustreError: 58148:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 2128.607216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2135.730638] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2144.698764] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2152.050351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.680635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.763674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2157.888315] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2158.048176] LustreError: 58174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2158.071920] LustreError: 58174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2163.185873] LustreError: 59175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.219356] LustreError: 59175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2167.459262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.992492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2173.356978] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2182.523474] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2189.482159] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2193.378772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2193.383651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.425384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2193.896958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.899331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2193.933659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2198.314788] Lustre: server umount lustre-OST0000 complete [ 2223.523082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.562858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.574193] Lustre: Skipped 1 previous similar message [ 2228.713846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.735699] Lustre: Skipped 1 previous similar message [ 2229.880456] Lustre: server umount lustre-MDT0000 complete [ 2233.825726] LustreError: 59180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.852941] LustreError: 59180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2233.896076] LustreError: 58155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078927 with bad export cookie 11918781286755195409 [ 2233.908818] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2233.913194] LustreError: 58155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2240.407518] Lustre: server umount lustre-MDT0001 complete [ 2254.152494] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2257.448897] Key type lgssc unregistered [ 2257.769200] LNet: 61869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2257.781707] LNetError: 61869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2257.815689] LNet: Removed LNI 192.168.204.145@tcp [ 2258.859979] Key type .llcrypt unregistered [ 2258.862235] Key type ._llcrypt unregistered [ 2286.802848] Key type ._llcrypt registered [ 2286.805791] Key type .llcrypt registered [ 2287.017525] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2288.450211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2288.637856] alg: No test for adler32 (adler32-zlib) [ 2289.738937] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2290.030641] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2291.752459] Key type lgssc registered [ 2293.021961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2302.182450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2302.233771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.850190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.965288] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2313.186356] LustreError: 63279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.202144] LustreError: 63279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2318.321064] LustreError: 63280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.667702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.007526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2325.048740] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2335.113287] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2342.216853] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2351.118675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2351.512424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2359.582936] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2368.098884] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2373.077204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2388.449356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2388.455627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.459474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2388.468804] Lustre: Skipped 1 previous similar message [ 2392.269542] Lustre: server umount lustre-OST0000 complete [ 2398.689206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.691401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2398.713270] Lustre: Skipped 1 previous similar message [ 2398.729904] Lustre: Skipped 2 previous similar messages [ 2403.676529] Lustre: server umount lustre-MDT0000 complete [ 2403.812492] LustreError: 63280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.843630] LustreError: 63280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2407.776523] LustreError: 63259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079101 with bad export cookie 7140260133124757574 [ 2407.782289] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2407.789700] LustreError: 63259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2408.235081] Lustre: server umount lustre-MDT0001 complete [ 2420.316758] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2424.730367] Key type lgssc unregistered [ 2425.202797] LNet: 66820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.221048] LNetError: 66820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.277994] LNet: Removed LNI 192.168.204.145@tcp [ 2427.746197] Key type .llcrypt unregistered [ 2427.747981] Key type ._llcrypt unregistered [ 2460.034552] Key type ._llcrypt registered [ 2460.039901] Key type .llcrypt registered [ 2460.199068] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2461.583557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2461.651851] alg: No test for adler32 (adler32-zlib) [ 2462.757459] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2463.119489] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2464.839181] Key type lgssc registered [ 2465.972805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2478.042406] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 11:19:29 (1785079169) [ 2486.564573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2486.583361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.213893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2494.251911] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2498.532406] LustreError: 68515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.548573] LustreError: 68515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2503.657524] LustreError: 68516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2505.972840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.631412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2514.365989] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2524.959630] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2532.290265] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2540.485584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2540.859176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.332321] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2555.475347] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2558.259158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2565.137604] Lustre: Failing over lustre-MDT0000 [ 2565.466832] Lustre: server umount lustre-MDT0000 complete [ 2566.327640] LustreError: 68510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.169988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2568.174792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2568.675857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2568.682564] LustreError: 68512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.696590] Lustre: Skipped 1 previous similar message [ 2568.708695] LustreError: 68512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2573.803491] LustreError: 68515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.822125] LustreError: 68515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2575.464749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.644538] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2575.809757] Lustre: *** cfs_fail_loc=135, val=0*** [ 2575.885748] LustreError: 71601:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2575.892646] LustreError: 71601:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.145@tcp: cfg command failed: rc = -2 [ 2575.899393] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2575.899393] [ 2575.905961] LustreError: MGC192.168.204.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2575.919503] LustreError: 71589:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2575.925780] LustreError: 71589:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2575.930280] LustreError: 71589:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2575.936889] LustreError: 71589:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2575.956133] Lustre: server umount lustre-MDT0000 complete [ 2575.958627] LustreError: 71589:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2582.732138] LustreError: 69801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.757139] LustreError: 69801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2584.897598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2585.055805] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2585.070108] LustreError: 68507:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2585.318098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.386659] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2587.784119] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2590.412852] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2590.752771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2590.841407] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2590.900128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2600.931634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.932569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2600.963877] Lustre: Skipped 1 previous similar message [ 2600.993915] Lustre: Skipped 1 previous similar message [ 2606.055568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2606.070456] Lustre: Skipped 1 previous similar message [ 2606.996651] Lustre: server umount lustre-OST0000 complete [ 2611.170263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.177913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2611.187759] Lustre: Skipped 1 previous similar message [ 2611.203426] Lustre: Skipped 1 previous similar message [ 2616.288533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2617.115681] Lustre: server umount lustre-MDT0000 complete [ 2621.409440] LustreError: 68510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.430952] LustreError: 68510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2621.460729] LustreError: 69247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079315 with bad export cookie 7304002259255687623 [ 2621.462914] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.470214] LustreError: 69247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2621.700152] Lustre: server umount lustre-MDT0001 complete [ 2634.114323] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2636.722290] Key type lgssc unregistered [ 2636.985236] LNet: 73374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2637.005385] LNetError: 73374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2638.067687] LNet: Removed LNI 192.168.204.145@tcp [ 2638.946846] Key type .llcrypt unregistered [ 2638.951927] Key type ._llcrypt unregistered [ 2664.688756] Key type ._llcrypt registered [ 2664.694428] Key type .llcrypt registered [ 2664.849788] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2666.104803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2666.164465] alg: No test for adler32 (adler32-zlib) [ 2667.183753] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2667.453959] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2669.199321] Key type lgssc registered [ 2670.688537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2680.211344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2680.241734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.968709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.173871] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2691.565762] LustreError: 74778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.605684] LustreError: 74778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2696.677249] LustreError: 74779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2697.298435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2697.954361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2703.145879] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2712.284629] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2718.684461] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2725.967199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2726.504044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.548750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2733.359584] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2742.274265] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2748.155780] Lustre: Failing over lustre-MDT0000 [ 2748.391374] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.422487] LustreError: 75768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.451792] LustreError: 75768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2748.635183] Lustre: server umount lustre-MDT0000 complete [ 2753.506805] LustreError: 74779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.538241] LustreError: 74779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2758.632077] LustreError: 74778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.646470] LustreError: 74778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2759.396463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.606761] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2759.823802] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (not set up) [ 2759.891524] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2760.065056] LustreError: 77856:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2760.074305] LustreError: 77856:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.145@tcp: cfg command failed: rc = -9 [ 2760.082935] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2760.082935] [ 2760.093976] LustreError: MGC192.168.204.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2760.111366] LustreError: 77844:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2760.123652] LustreError: 77844:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2760.132421] LustreError: 77844:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2760.144211] LustreError: 77844:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2760.187649] Lustre: server umount lustre-MDT0000 complete [ 2760.195159] LustreError: 77844:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2767.800440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.035736] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2768.052275] LustreError: 74771:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2768.059231] LustreError: 78202:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.204.145@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000005aaea2f0) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2768.077857] LustreError: 74089:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000983fd023 x1871791376072064/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2768.099177] LustreError: 74773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.128439] LustreError: 74773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2768.445903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.519937] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2770.057225] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2773.495287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2773.541367] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2773.566694] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2773.599569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2783.721361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.730989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2783.756153] Lustre: Skipped 3 previous similar messages [ 2783.786787] Lustre: Skipped 1 previous similar message [ 2788.834910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2789.529370] Lustre: server umount lustre-OST0000 complete [ 2793.953734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.957943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2793.972650] Lustre: Skipped 1 previous similar message [ 2793.995477] Lustre: Skipped 2 previous similar messages [ 2799.075602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2799.082475] Lustre: Skipped 1 previous similar message [ 2808.287302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2808.556264] Lustre: server umount lustre-MDT0000 complete [ 2809.315869] LustreError: 74778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.334366] LustreError: 74778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2812.440385] LustreError: 74760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079506 with bad export cookie 5799488510993625730 [ 2812.444365] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2812.446456] LustreError: 74760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2812.891228] Lustre: server umount lustre-MDT0001 complete [ 2826.344712] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2829.691831] Key type lgssc unregistered [ 2829.956859] LNet: 79632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2829.965356] LNetError: 79632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2829.988909] LNet: Removed LNI 192.168.204.145@tcp [ 2830.901133] Key type .llcrypt unregistered [ 2830.903662] Key type ._llcrypt unregistered [ 2845.845416] Key type ._llcrypt registered [ 2845.850745] Key type .llcrypt registered [ 2846.074498] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2846.480668] Key type .llcrypt unregistered [ 2846.490204] Key type ._llcrypt unregistered [ 2873.467195] Key type ._llcrypt registered [ 2873.473567] Key type .llcrypt registered [ 2873.636307] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2875.453469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2875.518870] alg: No test for adler32 (adler32-zlib) [ 2876.707541] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2877.022805] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2878.855186] Key type lgssc registered [ 2880.452432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2889.933753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2889.955507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2891.713235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.622847] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2900.964397] LustreError: 81528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.984360] LustreError: 81528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2906.084506] LustreError: 81529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.484584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.818097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2911.514945] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2919.234991] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2925.137777] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2931.616737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2931.990891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.122020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2938.582912] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2947.210769] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2961.890043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2961.893575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.907611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.933774] Lustre: Skipped 1 previous similar message [ 2967.020406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2967.024748] Lustre: Skipped 1 previous similar message [ 2967.305473] Lustre: server umount lustre-OST0000 complete [ 2972.129086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.133474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.149216] Lustre: Skipped 1 previous similar message [ 2972.160205] Lustre: Skipped 2 previous similar messages [ 2977.251509] LustreError: 81529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.351736] Lustre: server umount lustre-MDT0000 complete [ 2981.325128] LustreError: 81509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079675 with bad export cookie 5602530230528473089 [ 2981.337547] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2981.344652] LustreError: 81509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2981.611648] Lustre: server umount lustre-MDT0001 complete [ 2994.034399] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2997.340781] Key type lgssc unregistered [ 2997.636669] LNet: 85066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2997.643350] LNetError: 85066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2997.662386] LNet: Removed LNI 192.168.204.145@tcp [ 2998.705401] Key type .llcrypt unregistered [ 2998.707049] Key type ._llcrypt unregistered [ 3024.033875] Key type ._llcrypt registered [ 3024.036985] Key type .llcrypt registered [ 3024.206670] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3025.634354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3025.673647] alg: No test for adler32 (adler32-zlib) [ 3026.836885] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 3027.121580] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3028.855560] Key type lgssc registered [ 3030.037307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3038.776797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3038.798322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.372158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.410362] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3049.953924] LustreError: 86473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3049.974051] LustreError: 86473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3055.072471] LustreError: 86472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3056.287406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.673517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3061.630609] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3069.507708] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3075.724726] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3083.290203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3083.615452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.700655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 3090.662873] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3098.953597] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3106.278962] Lustre: server umount lustre-OST0000 complete [ 3111.403202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3111.415933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3111.419080] Lustre: Skipped 1 previous similar message [ 3111.432892] Lustre: Skipped 1 previous similar message [ 3116.512985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3116.523148] Lustre: Skipped 1 previous similar message [ 3120.710181] Lustre: server umount lustre-MDT0000 complete [ 3121.634754] LustreError: 87199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3121.672737] LustreError: 87199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3122.337698] Lustre: 85786:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785079800/real 1785079800] req@000000000081e750 x1871791753553664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785079816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3122.360779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.177044] LustreError: 86453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079819 with bad export cookie 11608262360230704899 [ 3125.179843] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3125.193381] LustreError: 86453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3125.237669] LustreError: 87201:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3125.264086] LustreError: 87201:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3125.945312] Lustre: server umount lustre-MDT0001 complete [ 3137.541760] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3141.624602] Key type lgssc unregistered [ 3142.161639] LNet: 89991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3142.189597] LNetError: 89991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3142.230140] LNet: Removed LNI 192.168.204.145@tcp [ 3143.445282] Key type .llcrypt unregistered [ 3143.449045] Key type ._llcrypt unregistered [ 3170.321657] Key type ._llcrypt registered [ 3170.326181] Key type .llcrypt registered [ 3170.459175] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3171.626634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3171.740777] alg: No test for adler32 (adler32-zlib) [ 3172.856956] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 3173.119819] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3174.799910] Key type lgssc registered [ 3176.254810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3184.806336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3184.829550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.233577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.039784] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3196.385317] LustreError: 91398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3196.406437] LustreError: 91398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3199.934110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.437515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3205.420911] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3213.647538] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3221.333886] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3229.803805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3230.117383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3231.158982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3238.560244] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3246.768871] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3261.922132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3261.930513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3261.947982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3263.193714] Lustre: server umount lustre-OST0000 complete [ 3271.144886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.150209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3271.157144] Lustre: Skipped 1 previous similar message [ 3273.689521] Lustre: server umount lustre-MDT0000 complete [ 3276.279695] LustreError: 91398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.302446] LustreError: 91398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3277.983875] LustreError: 92130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079972 with bad export cookie 16209395568775445139 [ 3277.985515] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3278.008182] LustreError: 92130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3288.693642] Lustre: server umount lustre-MDT0001 complete [ 3302.346833] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3305.102976] Key type lgssc unregistered [ 3305.425725] LNet: 94938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3305.438588] LNetError: 94938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3305.462475] LNet: Removed LNI 192.168.204.145@tcp [ 3306.322326] Key type .llcrypt unregistered [ 3306.324784] Key type ._llcrypt unregistered [ 3329.850565] Key type ._llcrypt registered [ 3329.854396] Key type .llcrypt registered [ 3329.979678] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3330.864231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3330.973061] alg: No test for adler32 (adler32-zlib) [ 3332.107900] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 3332.291903] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3333.983193] Key type lgssc registered [ 3335.285557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3343.962256] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3343.989288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3359.583410] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3364.703357] LustreError: 96320:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 3391.455902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.777840] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3405.067279] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3412.197866] Lustre: server umount lustre-OST0000 complete [ 3422.639979] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 11:35:13 (1785080113) [ 3430.691638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.067947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.260079] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3436.524778] LustreError: 97947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.570845] LustreError: 97947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3441.634882] LustreError: 97948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.763285] LustreError: 97947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.860440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3447.452342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3452.662864] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3461.974548] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3469.263278] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3477.194971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3477.660498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.357742] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3492.840383] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3501.259661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3516.909546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.922567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3516.932406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3522.025380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3522.036772] Lustre: Skipped 2 previous similar messages [ 3522.696574] Lustre: server umount lustre-OST0000 complete [ 3527.136913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3527.149121] Lustre: Skipped 1 previous similar message [ 3527.164345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3527.173972] Lustre: Skipped 1 previous similar message [ 3532.259904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3532.277902] Lustre: Skipped 1 previous similar message [ 3533.752444] Lustre: server umount lustre-MDT0000 complete [ 3537.376811] LustreError: 98673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.389246] LustreError: 98673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3537.944030] LustreError: 97929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080232 with bad export cookie 1333789230487495562 [ 3537.960624] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3537.961786] LustreError: 97929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3538.424425] Lustre: server umount lustre-MDT0001 complete [ 3550.762542] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3553.601394] Key type lgssc unregistered [ 3553.893534] LNet: 101644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3553.904500] LNetError: 101644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3554.923188] LNet: Removed LNI 192.168.204.145@tcp [ 3555.975295] Key type .llcrypt unregistered [ 3555.977276] Key type ._llcrypt unregistered [ 3601.311897] Key type ._llcrypt registered [ 3601.315223] Key type .llcrypt registered [ 3601.450072] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3602.822983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3602.875086] alg: No test for adler32 (adler32-zlib) [ 3603.922321] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 3604.195484] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3605.903472] Key type lgssc registered [ 3607.368428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3617.109793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3617.145155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.054600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3624.350637] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3629.545509] LustreError: 103823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3629.563178] LustreError: 103823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3634.164209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3634.696907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3640.136947] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3648.411186] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3655.636648] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3663.262859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3663.710673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3664.763812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3670.675246] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3679.861894] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3694.572229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.572946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3694.593786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.889741] Lustre: server umount lustre-OST0000 complete [ 3704.802443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.822755] Lustre: Skipped 1 previous similar message [ 3704.829842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3704.836937] Lustre: Skipped 2 previous similar messages [ 3708.608328] Lustre: server umount lustre-MDT0000 complete [ 3709.921558] LustreError: 103823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3709.952413] LustreError: 103823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3713.565925] LustreError: 103803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080407 with bad export cookie 3367875789807775391 [ 3713.575478] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3713.587890] LustreError: 103803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3714.029042] Lustre: server umount lustre-MDT0001 complete [ 3728.382023] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3731.845892] Key type lgssc unregistered [ 3732.229377] LNet: 107341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3732.255733] LNetError: 107341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3732.284816] LNet: Removed LNI 192.168.204.145@tcp [ 3733.298328] Key type .llcrypt unregistered [ 3733.307150] Key type ._llcrypt unregistered [ 3760.362630] Key type ._llcrypt registered [ 3760.366474] Key type .llcrypt registered [ 3760.714635] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3762.698511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3762.941640] alg: No test for adler32 (adler32-zlib) [ 3764.235953] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 3764.679790] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3766.343463] Key type lgssc registered [ 3768.078676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3776.119204] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3776.133439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3791.711442] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3796.834383] LustreError: 108582:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 3823.583461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3830.981228] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3839.690172] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3847.400850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3847.579735] LustreError: MGC192.168.204.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3847.586416] LustreError: 109598:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3847.590325] LustreError: 109598:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3847.596716] LustreError: 109598:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3847.684693] Lustre: server umount lustre-MDT0000 complete [ 3847.688649] LustreError: 109598:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3867.616448] Lustre: 107919:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785080545/real 1785080545] req@000000002885fce2 x1871792526331264/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785080561 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3867.658653] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3867.673217] LustreError: 108594:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3888.269655] Lustre: server umount lustre-OST0000 complete [ 3900.298716] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 3916.900441] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3962.363553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3962.651436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3962.692328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3962.831860] Lustre: lustre-MDT0000: new disk, initializing [ 3962.966848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3963.001256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3967.933877] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3980.539571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.704583] Lustre: 112404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3980.789482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3980.794255] Lustre: Skipped 1 previous similar message [ 3980.878345] Lustre: lustre-MDT0001: new disk, initializing [ 3981.034215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3981.091297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3981.104272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3986.638922] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3998.977645] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4005.879958] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4013.653984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4014.002487] Lustre: lustre-OST0000: new disk, initializing [ 4014.006971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4014.012643] Lustre: 114365:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4014.121979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4020.319359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4020.340300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4020.467387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4021.050836] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4032.076844] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4037.972539] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4038.276035] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4043.672476] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4043.929931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4046.314900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.321812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4046.351535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4046.361392] LustreError: Skipped 1 previous similar message [ 4046.373309] Lustre: Skipped 1 previous similar message [ 4051.425209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4051.439478] Lustre: Skipped 1 previous similar message [ 4052.559151] Lustre: server umount lustre-OST0000 complete [ 4058.084555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4058.101485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.124368] Lustre: Skipped 1 previous similar message [ 4058.137773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4061.668931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4061.670542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4063.775168] Lustre: server umount lustre-MDT0000 complete [ 4068.543358] LustreError: 114364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080762 with bad export cookie 14612690180913781129 [ 4068.553910] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4068.557706] LustreError: 114364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4069.004626] Lustre: server umount lustre-MDT0001 complete [ 4081.620798] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 11:46:12 (1785080772) [ 4086.239398] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4105.921188] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4117.838662] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4160.921336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.132310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4161.189051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4161.316446] Lustre: lustre-MDT0000: new disk, initializing [ 4161.446452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4161.498529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4168.380114] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4181.480239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.705263] Lustre: 119263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4181.821830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4181.831976] Lustre: Skipped 1 previous similar message [ 4181.994949] Lustre: lustre-MDT0001: new disk, initializing [ 4182.108151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4182.187989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4182.225994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4187.675664] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4199.055089] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4207.185137] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4214.509891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4214.755722] Lustre: lustre-OST0000: new disk, initializing [ 4214.758891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4214.767308] Lustre: 121221:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4214.905925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4220.007206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4220.022368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4220.154045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4222.105305] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4233.711154] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4239.485603] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4239.685456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4244.645896] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4244.870230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4251.111499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4251.115073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.130473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4251.138449] Lustre: Skipped 1 previous similar message [ 4251.154151] Lustre: Skipped 2 previous similar messages [ 4261.343347] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4261.355520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4261.364836] Lustre: Skipped 3 previous similar messages [ 4261.692781] Lustre: server umount lustre-OST0000 complete [ 4266.466034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.492256] Lustre: Skipped 1 previous similar message [ 4280.287111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4280.577671] Lustre: server umount lustre-MDT0000 complete [ 4281.825093] LustreError: 119273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4281.844246] LustreError: 119273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4284.978243] LustreError: 121220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080979 with bad export cookie 14612690180913782011 [ 4284.980049] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4284.997670] LustreError: 121220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4285.371572] Lustre: server umount lustre-MDT0001 complete [ 4294.257398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.754153] LustreError: 123435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4294.844089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4299.437380] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4300.261775] LustreError: 123436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.360579] LustreError: 123435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4309.473805] LustreError: 123436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4309.732766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.154550] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4323.941087] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4329.368600] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4337.184211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4345.068438] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4353.481241] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4361.481043] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4368.259672] Lustre: server umount lustre-OST0000 complete [ 4375.529822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4375.542470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4375.542862] Lustre: Skipped 1 previous similar message [ 4375.564604] Lustre: Skipped 7 previous similar messages [ 4386.784167] Lustre: 107921:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785081064/real 1785081064] req@00000000f70cce65 x1871792526453120/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785081080 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4386.810115] Lustre: 107921:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4387.295170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4387.644788] Lustre: server umount lustre-MDT0000 complete [ 4391.871803] LustreError: 123416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081085 with bad export cookie 14612690180913782893 [ 4391.884819] LustreError: 123416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4391.887497] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4392.233243] Lustre: server umount lustre-MDT0001 complete [ 4404.294841] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4407.611063] Key type lgssc unregistered [ 4407.962821] LNet: 127049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4407.971034] LNetError: 127049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4407.985425] LNet: Removed LNI 192.168.204.145@tcp [ 4408.942372] Key type .llcrypt unregistered [ 4408.945540] Key type ._llcrypt unregistered [ 4428.783865] Key type ._llcrypt registered [ 4428.787615] Key type .llcrypt registered [ 4428.936991] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4444.972737] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4446.766422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4446.823595] alg: No test for adler32 (adler32-zlib) [ 4447.922027] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 4448.195335] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4449.904683] Key type lgssc registered [ 4451.308839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4490.341679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4490.405947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.742289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4491.788669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4491.914352] Lustre: lustre-MDT0000: new disk, initializing [ 4492.071386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4492.117180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4497.468580] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4509.541629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.684893] Lustre: 130192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4509.787917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4509.792187] Lustre: Skipped 1 previous similar message [ 4509.910060] Lustre: lustre-MDT0001: new disk, initializing [ 4510.067640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4510.113507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4510.126973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4515.595632] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4525.952705] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4532.174888] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4539.522782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4539.724825] Lustre: lustre-OST0000: new disk, initializing [ 4539.728552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4539.733514] Lustre: 132155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4539.824711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4545.677396] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4547.152704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4547.158865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4547.257504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4555.584839] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4560.894181] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4561.161847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4565.990925] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4566.242526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4573.152440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4573.158044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.166286] LustreError: Skipped 1 previous similar message [ 4573.167688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4573.179539] Lustre: Skipped 1 previous similar message [ 4574.726182] Lustre: server umount lustre-OST0000 complete [ 4581.858082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4581.865450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.885715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4581.891229] Lustre: Skipped 1 previous similar message [ 4583.393395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4583.393586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4584.970107] Lustre: server umount lustre-MDT0000 complete [ 4588.941354] LustreError: 130184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081283 with bad export cookie 7736327975791292784 [ 4588.950717] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4588.958614] LustreError: 130184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4589.429710] Lustre: server umount lustre-MDT0001 complete [ 4600.192685] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 11:54:51 (1785081291) [ 4607.995250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.450568] LustreError: 134558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4608.468329] LustreError: 134558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4608.569853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4613.602387] LustreError: 134559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4614.126750] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4618.728505] LustreError: 134558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4623.854566] LustreError: 134558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4623.949648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.511090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4629.270073] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4639.972354] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4647.085736] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4649.962069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4649.972807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4650.008463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4650.019603] Lustre: Skipped 1 previous similar message [ 4654.560334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4654.563897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4654.601108] Lustre: Skipped 1 previous similar message [ 4664.287144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4664.643304] Lustre: server umount lustre-MDT0000 complete [ 4664.816612] LustreError: 134559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.837951] LustreError: 134559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4669.140680] LustreError: 134539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081363 with bad export cookie 7736327975791293666 [ 4669.159670] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.591598] Lustre: server umount lustre-MDT0001 complete [ 4680.923947] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 11:56:11 (1785081371) [ 4688.871527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4704.481313] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4709.600246] LustreError: 136963:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 4735.328108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4741.589146] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4749.917861] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4752.404994] Lustre: server umount lustre-OST0000 complete [ 4762.305846] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 11:57:33 (1785081453) [ 4770.156890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.716556] LustreError: 138397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.742099] LustreError: 138397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4770.830201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4775.876375] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4784.728270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.339358] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4797.642343] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4804.344911] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4811.480234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4811.918065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.932745] Lustre: Skipped 1 previous similar message [ 4817.510950] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4826.067180] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4838.925805] Lustre: server umount lustre-MDT0000 complete [ 4840.930240] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4840.939405] LustreError: 139759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4840.942685] Lustre: Skipped 1 previous similar message [ 4840.961354] LustreError: 139759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4842.878674] LustreError: 139131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081536 with bad export cookie 7736327975791294331 [ 4842.900523] LustreError: 139131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4842.905573] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4843.262972] Lustre: server umount lustre-MDT0001 complete [ 4857.533630] Lustre: server umount lustre-OST0000 complete [ 4867.329655] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 11:59:18 (1785081558) [ 4874.642655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4875.224956] LustreError: 142061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4875.252720] LustreError: 142061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4880.281907] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4889.056403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4889.469842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4889.477726] Lustre: Skipped 1 previous similar message [ 4893.620505] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4901.087794] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4906.499486] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4913.442421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4920.179112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4920.651605] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4928.608763] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4933.950317] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4934.182138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4937.984735] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4938.180675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4940.770741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.774086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4940.781758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4940.800430] Lustre: Skipped 2 previous similar messages [ 4940.826886] Lustre: Skipped 3 previous similar messages [ 4946.571735] Lustre: server umount lustre-OST0000 complete [ 4951.009520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4951.017120] LustreError: Skipped 1 previous similar message [ 4961.249229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4961.260469] Lustre: Skipped 7 previous similar messages [ 4964.319256] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4964.458817] Lustre: server umount lustre-MDT0000 complete [ 4966.371907] LustreError: 142061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4966.388389] LustreError: 142061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4968.990280] LustreError: 143897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081663 with bad export cookie 7736327975791295395 [ 4968.993910] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4968.999289] LustreError: 143897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4969.447710] Lustre: server umount lustre-MDT0001 complete [ 4979.810525] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 12:01:11 (1785081671) [ 4988.430646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5004.137299] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5009.247535] LustreError: 146218:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 5034.975310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5034.987602] Lustre: Skipped 1 previous similar message [ 5042.409457] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5050.910343] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5058.845283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.473039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5064.252540] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5072.464822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5077.291557] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5085.355929] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5090.758186] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5095.573776] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5095.812986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5100.951179] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5101.197745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5103.585264] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.587598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5103.615928] Lustre: Skipped 3 previous similar messages [ 5103.646692] Lustre: Skipped 2 previous similar messages [ 5109.770144] Lustre: server umount lustre-MDT0000 complete [ 5113.831150] LustreError: 147243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.869944] LustreError: 147243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5114.244520] LustreError: 146226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081808 with bad export cookie 7736327975791296424 [ 5114.249268] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5114.255717] LustreError: 146226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5114.730566] Lustre: server umount lustre-MDT0001 complete [ 5129.488678] Lustre: server umount lustre-OST0000 complete [ 5137.239867] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 12:03:48 (1785081828) [ 5144.883363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5160.479899] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5165.599634] LustreError: 150390:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 5197.847568] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5204.607372] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5212.753240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.229319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5218.628477] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5229.215229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5233.790895] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5241.718472] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5247.565827] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5254.462972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5254.637733] Lustre: lustre-OST0001: new disk, initializing [ 5254.641810] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5254.647839] Lustre: 153249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5261.071570] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5261.361466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5261.373567] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5261.517281] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5271.807796] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5277.266334] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5277.597574] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5282.813146] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5283.080509] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5285.859605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5285.876717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.900372] Lustre: Skipped 1 previous similar message [ 5285.902947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5285.911525] Lustre: Skipped 3 previous similar messages [ 5291.707715] Lustre: server umount lustre-OST0000 complete [ 5302.769185] Lustre: server umount lustre-OST0001 complete [ 5321.184817] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5321.446446] Lustre: server umount lustre-MDT0000 complete [ 5325.281352] LustreError: 150397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082019 with bad export cookie 7736327975791297376 [ 5325.292789] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5325.296938] LustreError: 150397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5325.824079] Lustre: server umount lustre-MDT0001 complete [ 5351.109698] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 12:07:22 (1785082042) [ 5352.843845] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5354.507755] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5356.766732] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 12:07:28 (1785082048) [ 5363.381300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.626455] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5363.650309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5363.871946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.886610] Lustre: Skipped 6 previous similar messages [ 5368.956617] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5377.558993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5377.676112] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5377.718458] Lustre: 156798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5382.287157] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5390.282170] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5396.778527] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5404.625406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5404.778918] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5410.994095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5411.917968] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5420.717773] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5425.834248] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5426.127662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5431.049044] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5431.292591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5436.386790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5436.400388] LustreError: Skipped 2 previous similar messages [ 5436.411195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.421837] Lustre: Skipped 5 previous similar messages [ 5436.429221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5436.431495] Lustre: Skipped 10 previous similar messages [ 5439.502975] Lustre: server umount lustre-OST0000 complete [ 5462.087327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5468.671110] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5475.951306] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5477.811797] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.852973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.854358] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5477.886424] Lustre: Skipped 1 previous similar message [ 5481.615184] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5481.789905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5486.336103] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5486.564924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5493.145274] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f5d88406800.ost_server_uuid 50 [ 5495.043709] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f5d88406800.ost_server_uuid in FULL state after 0 sec [ 5495.309693] LustreError: 158352:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.45@tcp) returned error from blocking AST (req@00000000dd517574 x1871793243817984 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000abbe115c/0x6b5cf54e1ef50cba lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.45@tcp remote: 0xf6b0d63010a0cb7e expref: 5 pid: 156804 timeout: 5595 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5495.351725] LustreError: lustre-MDT0000: A client on nid 192.168.204.45@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5495.369236] LustreError: 156794:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.45@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000abbe115c/0x6b5cf54e1ef50cba lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.45@tcp remote: 0xf6b0d63010a0cb7e expref: 6 pid: 156804 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5500.908617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5500.924988] LustreError: Skipped 1 previous similar message [ 5506.183507] Lustre: server umount lustre-OST0000 complete [ 5516.706096] Lustre: server umount lustre-MDT0000 complete [ 5518.817094] LustreError: 158353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.825727] LustreError: 158353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 5520.791845] LustreError: 158653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082214 with bad export cookie 7736327975791298559 [ 5520.801363] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5520.813426] LustreError: 158653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5535.199294] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5535.479984] Lustre: server umount lustre-MDT0001 complete [ 5548.017672] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5551.390820] Key type lgssc unregistered [ 5551.783707] LNet: 162889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5551.813533] LNetError: 162889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5552.873845] LNet: Removed LNI 192.168.204.145@tcp [ 5553.896334] Key type .llcrypt unregistered [ 5553.898837] Key type ._llcrypt unregistered [ 5581.484076] Key type ._llcrypt registered [ 5581.487365] Key type .llcrypt registered [ 5581.628902] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5582.826098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5583.048628] alg: No test for adler32 (adler32-zlib) [ 5584.300556] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 5584.738536] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5586.567273] Key type lgssc registered [ 5588.107449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5596.932667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5596.977445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.439705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5602.819705] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5608.945831] LustreError: 164295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5608.964234] LustreError: 164295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5611.564643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5612.056385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5616.586730] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5625.195222] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5631.993497] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5639.729284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5640.153903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5644.273466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5647.081953] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5655.262093] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5664.544037] Lustre: Failing over lustre-MDT0000 [ 5664.919666] Lustre: server umount lustre-MDT0000 complete [ 5667.815037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.817185] LustreError: 164291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.844346] Lustre: Skipped 2 previous similar messages [ 5667.871727] LustreError: 164291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5672.934225] LustreError: 164295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5672.961084] LustreError: 164295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5677.076343] LustreError: 164274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.45@tcp arrived at 1785082371 with bad export cookie 6358047120213257845 [ 5677.088834] LustreError: 164274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5678.056416] LustreError: 165723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5678.082510] LustreError: 165723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5683.167216] Lustre: 163608:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785082361/real 1785082361] req@000000002441a7a0 x1871794434764672/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785082377 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5683.181997] LustreError: 164294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5683.195173] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5683.235214] LustreError: 164294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5684.073973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5693.408908] LustreError: 165984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.415269] LustreError: 163607:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e7bf9333 x1871794434770176/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5693.447063] LustreError: 165984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5693.746089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5693.798660] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5698.998906] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5699.046405] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5699.072570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5701.471790] Lustre: lustre-MDT0000: Denying connection for new client 196a45b6-7685-4c66-a95f-1827a2f21840 (at 192.168.204.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5713.888833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5713.894316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.912261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5714.418280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5716.217376] Lustre: server umount lustre-OST0000 complete [ 5720.377276] LustreError: 168007:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5720.384608] Lustre: 167360:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5720.391696] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5720.403189] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5720.410536] LustreError: 167360:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006267bd34 x1871794434785152/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5720.432791] LustreError: 167360:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5720.443331] LustreError: 167360:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5720.450679] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5720.730658] Lustre: server umount lustre-MDT0000 complete [ 5724.643306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.671623] Lustre: Skipped 1 previous similar message [ 5724.675121] LustreError: 164291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5724.866335] LustreError: 164275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082418 with bad export cookie 6358047120213257950 [ 5724.871744] LustreError: 163607:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5724.872776] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5724.879103] LustreError: 164275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5727.250955] Lustre: server umount lustre-MDT0001 complete [ 5741.365842] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5745.286321] Key type lgssc unregistered [ 5745.607623] LNet: 168756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5745.620311] LNetError: 168756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5745.665862] LNet: Removed LNI 192.168.204.145@tcp [ 5746.578187] Key type .llcrypt unregistered [ 5746.580716] Key type ._llcrypt unregistered [ 5773.399041] Key type ._llcrypt registered [ 5773.400686] Key type .llcrypt registered [ 5773.522266] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5774.622923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5774.685574] alg: No test for adler32 (adler32-zlib) [ 5775.698503] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 5775.989316] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5777.655465] Key type lgssc registered [ 5778.897527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5786.798211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5786.840475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.316460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.686789] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5798.884757] LustreError: 170166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.907607] LustreError: 170166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5800.644440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.009140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5804.986571] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5812.238799] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5817.673601] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5823.328757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5823.624969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5829.046256] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5833.909372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5838.131535] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5847.521499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5847.532491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.550575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5847.745732] Lustre: server umount lustre-OST0000 complete [ 5854.691174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.711321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5854.716133] Lustre: Skipped 1 previous similar message [ 5859.809986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5859.814200] Lustre: Skipped 1 previous similar message [ 5863.111807] Lustre: server umount lustre-MDT0000 complete [ 5864.932355] LustreError: 171654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.961432] LustreError: 171654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5866.928783] LustreError: 172650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082561 with bad export cookie 10008510034860381966 [ 5866.949591] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5866.954137] LustreError: 172650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5867.336112] Lustre: server umount lustre-MDT0001 complete [ 5879.652588] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5883.570927] Key type lgssc unregistered [ 5884.047192] LNet: 173685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5884.063894] LNetError: 173685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5885.094207] LNet: Removed LNI 192.168.204.145@tcp [ 5886.061741] Key type .llcrypt unregistered [ 5886.069185] Key type ._llcrypt unregistered [ 5924.444683] Key type ._llcrypt registered [ 5924.446863] Key type .llcrypt registered [ 5924.534297] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5925.933793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5925.981165] alg: No test for adler32 (adler32-zlib) [ 5927.284057] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 5927.629110] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5929.335284] Key type lgssc registered [ 5930.618800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5938.937492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5938.956241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.400730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5944.869417] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5950.949101] LustreError: 175379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5950.979453] LustreError: 175379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5953.722381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5954.079848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5958.888741] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5967.470461] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5973.957813] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5981.579421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5982.029497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5986.163091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5989.476728] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5997.227322] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6006.760339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6006.773079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.789047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6009.838503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6009.843492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6009.859674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6009.892695] Lustre: Skipped 1 previous similar message [ 6011.193900] Lustre: server umount lustre-OST0000 complete [ 6015.456477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6015.464495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6015.490229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6020.069539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6020.070370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6021.609763] Lustre: server umount lustre-MDT0000 complete [ 6025.185587] LustreError: 175373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6025.221721] LustreError: 175373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6025.574279] LustreError: 175359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082719 with bad export cookie 12015421809036637688 [ 6025.580303] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6025.589288] LustreError: 175359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6026.182436] Lustre: server umount lustre-MDT0001 complete [ 6037.727033] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6040.880148] Key type lgssc unregistered [ 6041.169067] LNet: 178896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6041.175164] LNetError: 178896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6041.193321] LNet: Removed LNI 192.168.204.145@tcp [ 6042.110960] Key type .llcrypt unregistered [ 6042.115995] Key type ._llcrypt unregistered [ 6067.352318] Key type ._llcrypt registered [ 6067.356290] Key type .llcrypt registered [ 6067.524702] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6068.833825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6068.900753] alg: No test for adler32 (adler32-zlib) [ 6070.101517] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6070.423082] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6072.167271] Key type lgssc registered [ 6073.694561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6087.675324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6087.701603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.190107] Lustre: *** cfs_fail_loc=135, val=0*** [ 6089.344353] LustreError: 180413:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6089.362211] LustreError: 180413:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.145@tcp: cfg command failed: rc = -2 [ 6089.378116] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6089.378116] [ 6089.396217] LustreError: MGC192.168.204.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6089.431723] LustreError: 180374:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6089.445232] LustreError: 180374:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6089.456404] LustreError: 180374:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6089.467202] LustreError: 180374:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6089.610646] Lustre: server umount lustre-MDT0000 complete [ 6089.621489] LustreError: 180374:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6097.567795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6098.127944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.859937] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6108.646091] LustreError: 180784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.660088] LustreError: 180784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6113.047305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6113.586566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6118.644001] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6126.921572] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6132.910183] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6138.337988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6138.355218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6138.367790] Lustre: Skipped 1 previous similar message [ 6139.368390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6139.372109] Lustre: Skipped 1 previous similar message [ 6141.873320] Lustre: server umount lustre-MDT0000 complete [ 6144.481613] LustreError: 181510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6144.498499] LustreError: 181510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6146.656703] LustreError: 180765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082840 with bad export cookie 3763096968896419318 [ 6146.666934] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6146.677761] LustreError: 180765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6147.076995] Lustre: server umount lustre-MDT0001 complete [ 6161.032471] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6164.209389] Key type lgssc unregistered [ 6164.561544] LNet: 183097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6164.578872] LNetError: 183097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6164.632934] LNet: Removed LNI 192.168.204.145@tcp [ 6165.709329] Key type .llcrypt unregistered [ 6165.713106] Key type ._llcrypt unregistered [ 6189.544994] Key type ._llcrypt registered [ 6189.549843] Key type .llcrypt registered [ 6189.684821] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6190.037329] Key type .llcrypt unregistered [ 6190.039885] Key type ._llcrypt unregistered [ 6206.716536] Key type ._llcrypt registered [ 6206.719836] Key type .llcrypt registered [ 6206.806577] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6208.166929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6208.190786] alg: No test for adler32 (adler32-zlib) [ 6209.400944] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6209.680787] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6211.383647] Key type lgssc registered [ 6212.640006] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6220.333063] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6220.361046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6235.872417] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6240.992900] LustreError: 185222:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 6267.679663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6274.061979] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6281.441202] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6288.179244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.728732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6288.801503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6293.197767] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6293.990795] LustreError: 185248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6294.027860] LustreError: 185248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6299.118222] LustreError: 185247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6301.468815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6301.836117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6306.678541] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6314.504482] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6319.224075] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6321.891996] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6334.946198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6334.949911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6334.970415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6337.505293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6337.517068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6337.534616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6339.051718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6339.071912] Lustre: Skipped 1 previous similar message [ 6340.437920] Lustre: server umount lustre-OST0000 complete [ 6347.755682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6347.767481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6347.790105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6351.041196] Lustre: server umount lustre-MDT0000 complete [ 6354.411334] LustreError: 186255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.253028] LustreError: 185229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083049 with bad export cookie 8400873699298691785 [ 6355.256911] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6355.264973] LustreError: 185229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6355.896123] Lustre: server umount lustre-MDT0001 complete [ 6369.024882] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6372.421777] Key type lgssc unregistered [ 6372.758801] LNet: 189214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6372.775635] LNetError: 189214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6372.790140] LNet: Removed LNI 192.168.204.145@tcp [ 6373.759333] Key type .llcrypt unregistered [ 6373.763560] Key type ._llcrypt unregistered [ 6399.336995] Key type ._llcrypt registered [ 6399.341065] Key type .llcrypt registered [ 6399.441402] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6400.558852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6400.626114] alg: No test for adler32 (adler32-zlib) [ 6401.650975] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6401.956468] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6403.639143] Key type lgssc registered [ 6404.782881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6412.163506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6412.183317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6413.622113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6418.190608] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6423.010966] LustreError: 190622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6423.025843] LustreError: 190622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6426.928031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6427.407931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6432.158279] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6440.132929] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6446.570078] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6454.065870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6454.525830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6458.616126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6461.301906] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6470.000775] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6474.970503] Lustre: Failing over lustre-MDT0000 [ 6475.314831] Lustre: server umount lustre-MDT0000 complete [ 6476.999660] LustreError: 192638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6478.316190] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.338989] Lustre: Skipped 1 previous similar message [ 6478.344726] LustreError: 191628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6478.369913] LustreError: 191628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6482.064645] LustreError: 190618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.177728] LustreError: 192638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.210870] LustreError: 192638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6494.687161] Lustre: 189935:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785083172/real 1785083172] req@00000000116937d6 x1871795292497280/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785083188 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6494.732453] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6495.267352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6497.416332] LustreError: 190619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6497.438229] LustreError: 190619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6505.300021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6505.363323] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6508.696851] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6509.970881] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6510.578656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6510.614898] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6510.676130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6514.218208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6523.309509] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6536.674232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6536.679561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6536.689347] Lustre: Skipped 1 previous similar message [ 6536.693967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6538.721454] Lustre: server umount lustre-OST0000 complete [ 6546.405072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6546.412071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6546.418195] Lustre: Skipped 1 previous similar message [ 6546.437179] Lustre: Skipped 1 previous similar message [ 6549.097536] Lustre: server umount lustre-MDT0000 complete [ 6551.520670] LustreError: 191628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6551.546433] LustreError: 191628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6552.882841] LustreError: 194490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083246 with bad export cookie 1741535952111992746 [ 6552.883840] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6552.894418] LustreError: 194490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6562.500701] Lustre: server umount lustre-MDT0001 complete [ 6573.913702] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6577.098707] Key type lgssc unregistered [ 6577.447456] LNet: 195562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6577.459277] LNetError: 195562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6577.486139] LNet: Removed LNI 192.168.204.145@tcp [ 6578.271246] Key type .llcrypt unregistered [ 6578.273134] Key type ._llcrypt unregistered [ 6608.629664] Key type ._llcrypt registered [ 6608.633498] Key type .llcrypt registered [ 6608.814823] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6610.067926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6610.250816] alg: No test for adler32 (adler32-zlib) [ 6611.323363] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6611.541429] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6613.259828] Key type lgssc registered [ 6614.742255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6625.389903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6625.438163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.097779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6632.530729] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6637.546410] LustreError: 196970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6637.568312] LustreError: 196970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6642.210768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6642.774504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6647.357662] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6655.951869] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6661.894085] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6668.751077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6669.452918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6672.487883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6677.284555] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6686.843598] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6691.107819] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6699.214756] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6718.096205] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6729.190975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6729.203644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.257633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6729.269587] Lustre: Skipped 1 previous similar message [ 6733.580510] Lustre: server umount lustre-OST0000 complete [ 6739.424660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.427165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6739.436478] Lustre: Skipped 2 previous similar messages [ 6743.769410] Lustre: server umount lustre-MDT0000 complete [ 6744.544223] LustreError: 196969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6744.581651] LustreError: 196969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6748.595545] LustreError: 196951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083442 with bad export cookie 11959888364694435062 [ 6748.610637] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6748.611045] LustreError: 196951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6748.977849] Lustre: server umount lustre-MDT0001 complete [ 6760.607522] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6763.364408] Key type lgssc unregistered [ 6763.708547] LNet: 200827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6763.724843] LNetError: 200827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6763.757375] LNet: Removed LNI 192.168.204.145@tcp [ 6764.528101] Key type .llcrypt unregistered [ 6764.530117] Key type ._llcrypt unregistered [ 6785.110960] Key type ._llcrypt registered [ 6785.112888] Key type .llcrypt registered [ 6785.242719] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6786.304615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6786.337381] alg: No test for adler32 (adler32-zlib) [ 6787.626818] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6787.885669] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6789.520150] Key type lgssc registered [ 6790.733411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6799.166379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6799.187309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.685186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6806.704459] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6810.083652] LustreError: 202090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.107566] LustreError: 202090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6815.004305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6815.205204] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6815.333592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6819.817382] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6827.916032] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6834.748863] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6841.876986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6842.320170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6845.426443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6849.256827] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6858.007926] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6864.618310] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6875.577430] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6883.904161] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6892.044821] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6901.735451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6901.743845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6901.746872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6901.762093] Lustre: Skipped 1 previous similar message [ 6901.783507] LustreError: Skipped 1 previous similar message [ 6906.855294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6906.867611] Lustre: Skipped 2 previous similar messages [ 6907.985219] Lustre: server umount lustre-OST0000 complete [ 6912.481038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6912.497284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6912.519127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6917.088594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6917.090256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6918.670681] Lustre: server umount lustre-MDT0000 complete [ 6922.215091] LustreError: 203093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.251155] LustreError: 203093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6923.129615] LustreError: 202821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083617 with bad export cookie 10729211755697833219 [ 6923.130858] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6923.147768] LustreError: 202821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6923.473759] Lustre: server umount lustre-MDT0001 complete [ 6936.698872] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6939.510296] Key type lgssc unregistered [ 6939.888540] LNet: 206996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6939.894445] LNetError: 206996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6939.915093] LNet: Removed LNI 192.168.204.145@tcp [ 6940.870933] Key type .llcrypt unregistered [ 6940.874113] Key type ._llcrypt unregistered [ 6966.178637] Key type ._llcrypt registered [ 6966.182262] Key type .llcrypt registered [ 6966.286273] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6967.254792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6967.395488] alg: No test for adler32 (adler32-zlib) [ 6968.488512] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6968.789376] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6970.455801] Key type lgssc registered [ 6972.011422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6980.522704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6980.544734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6982.071540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6987.044520] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6992.356878] LustreError: 208405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6992.373510] LustreError: 208405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6996.661707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6997.089725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7001.951185] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7010.815682] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7018.907105] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7027.389109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7027.786959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7032.815647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7035.168584] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7044.729368] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7067.619376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7067.624469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7067.685382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7070.007408] Lustre: server umount lustre-OST0000 complete [ 7077.859174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.879451] Lustre: Skipped 1 previous similar message [ 7077.893565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7077.907323] Lustre: Skipped 2 previous similar messages [ 7082.982051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7082.997850] Lustre: Skipped 1 previous similar message [ 7087.074667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7087.082873] Lustre: Skipped 1 previous similar message [ 7089.121372] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7089.276328] Lustre: server umount lustre-MDT0000 complete [ 7092.214786] LustreError: 208404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7092.256974] LustreError: 208404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7094.062872] LustreError: 208385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083788 with bad export cookie 6300263573997239716 [ 7094.074996] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7094.081317] LustreError: 208385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7094.549817] Lustre: server umount lustre-MDT0001 complete [ 7107.322903] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7109.937787] Key type lgssc unregistered [ 7110.166080] LNet: 212074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7110.170930] LNetError: 212074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7110.186665] LNet: Removed LNI 192.168.204.145@tcp [ 7110.957213] Key type .llcrypt unregistered [ 7110.960149] Key type ._llcrypt unregistered [ 7131.850664] Key type ._llcrypt registered [ 7131.852812] Key type .llcrypt registered [ 7131.962110] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7133.341964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7133.456496] alg: No test for adler32 (adler32-zlib) [ 7134.709537] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 7134.931243] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7136.624650] Key type lgssc registered [ 7138.142182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7147.270664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7147.283435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7148.987460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7153.783641] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7158.247380] LustreError: 213342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7158.278686] LustreError: 213342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7163.024807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.345885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7168.239227] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7177.658704] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7185.186558] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7193.481904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7193.931496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7202.646274] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7213.218658] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7218.494348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7239.140449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7239.140861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7239.149791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7239.163166] Lustre: Skipped 1 previous similar message [ 7239.173187] Lustre: Skipped 1 previous similar message [ 7243.254431] Lustre: server umount lustre-OST0000 complete [ 7249.376609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7249.383051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7249.388623] Lustre: Skipped 1 previous similar message [ 7253.832413] Lustre: server umount lustre-MDT0000 complete [ 7254.497766] LustreError: 213342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7254.525304] LustreError: 213342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7257.578641] LustreError: 213322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083951 with bad export cookie 10322358882769798784 [ 7257.589882] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7257.601380] LustreError: 213322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7257.883291] Lustre: server umount lustre-MDT0001 complete [ 7270.581098] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7273.332991] Key type lgssc unregistered [ 7273.665916] LNet: 217031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7273.676326] LNetError: 217031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7273.709477] LNet: Removed LNI 192.168.204.145@tcp [ 7274.715439] Key type .llcrypt unregistered [ 7274.719160] Key type ._llcrypt unregistered [ 7296.332677] Key type ._llcrypt registered [ 7296.339128] Key type .llcrypt registered [ 7296.549437] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7298.056873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7298.123549] alg: No test for adler32 (adler32-zlib) [ 7299.277108] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 7299.597316] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7301.319187] Key type lgssc registered [ 7302.327477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7309.656334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7309.668250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7311.111045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7315.534724] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7321.578293] LustreError: 218294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7321.597174] LustreError: 218294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7325.591258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.977666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7331.169681] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7340.374515] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7347.971075] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7355.825436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7356.237942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.299303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7362.622973] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7371.093867] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7381.035404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7381.085029] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7381.389972] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7388.045298] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7397.239595] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7399.086524] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7411.306243] Lustre: Permanently deactivating lustre-OST0001 [ 7411.325957] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7417.956497] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7426.201699] Lustre: Permanently reactivating lustre-OST0001 [ 7426.234785] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7426.254122] Lustre: Skipped 2 previous similar messages [ 7427.866026] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7427.882524] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7427.902520] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7427.925493] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7427.982981] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7443.431503] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7443.437070] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7443.453436] Lustre: Skipped 1 previous similar message [ 7443.464542] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7443.470271] Lustre: Skipped 1 previous similar message [ 7447.567664] Lustre: server umount lustre-OST0001 complete [ 7448.553268] LustreError: 220110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7448.579831] LustreError: 220110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7452.641272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7452.650979] LustreError: Skipped 1 previous similar message [ 7452.660335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7452.679875] Lustre: Skipped 1 previous similar message [ 7452.682538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7453.677669] LustreError: 220108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.690960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7453.703951] LustreError: 220108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7453.737652] Lustre: Skipped 1 previous similar message [ 7458.664602] Lustre: server umount lustre-OST0000 complete [ 7463.906109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.910486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7463.913808] Lustre: Skipped 2 previous similar messages [ 7463.927412] Lustre: Skipped 1 previous similar message [ 7468.843449] Lustre: server umount lustre-MDT0000 complete [ 7469.026886] LustreError: 218295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7472.152162] LustreError: 221901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084166 with bad export cookie 6168486216667516873 [ 7472.159518] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7472.165421] LustreError: 221901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7472.515692] Lustre: server umount lustre-MDT0001 complete [ 7484.656236] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7488.295040] Key type lgssc unregistered [ 7488.714375] LNet: 223339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7488.724436] LNetError: 223339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7488.753133] LNet: Removed LNI 192.168.204.145@tcp [ 7489.656739] Key type .llcrypt unregistered [ 7489.659128] Key type ._llcrypt unregistered [ 7539.333916] Key type ._llcrypt registered [ 7539.338752] Key type .llcrypt registered [ 7539.498352] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7540.917763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7540.983568] alg: No test for adler32 (adler32-zlib) [ 7542.227949] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 7542.543991] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7544.327307] Key type lgssc registered [ 7545.609596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7556.075184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7556.124134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7557.467409] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7557.483304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7557.616855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7562.755778] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7572.902965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.135848] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7573.188401] Lustre: 225534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7573.583777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7577.937962] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7588.105744] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7594.234253] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7601.746802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7602.013839] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7602.199671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7608.535607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7608.773459] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7617.458258] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7621.704241] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7628.600426] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7633.358349] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7641.318413] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7651.792199] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7670.766462] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7670.773908] Lustre: Skipped 2 previous similar messages [ 7703.720130] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7703.735636] Lustre: Skipped 3 previous similar messages [ 7774.530489] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7774.540639] Lustre: Skipped 8 previous similar messages [ 7793.134267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7793.136325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7793.150397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7793.155257] Lustre: Skipped 1 previous similar message [ 7797.113318] Lustre: server umount lustre-OST0000 complete [ 7803.360844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7803.377791] Lustre: Skipped 2 previous similar messages [ 7803.378445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7808.485507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7808.487834] Lustre: Skipped 1 previous similar message [ 7813.601592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7813.612095] Lustre: Skipped 2 previous similar messages [ 7816.163043] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7816.396894] Lustre: server umount lustre-MDT0000 complete [ 7821.162702] LustreError: 227393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084515 with bad export cookie 7687324546032171061 [ 7821.168538] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7821.187190] LustreError: 227393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7821.651453] Lustre: server umount lustre-MDT0001 complete [ 7833.628363] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7836.406569] Key type lgssc unregistered [ 7836.677112] LNet: 230290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7836.688648] LNetError: 230290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7836.719157] LNet: Removed LNI 192.168.204.145@tcp [ 7837.408653] Key type .llcrypt unregistered [ 7837.413706] Key type ._llcrypt unregistered [ 7862.624103] Key type ._llcrypt registered [ 7862.625850] Key type .llcrypt registered [ 7862.723508] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7863.642980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7863.743845] alg: No test for adler32 (adler32-zlib) [ 7864.892612] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 7865.178968] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7866.871359] Key type lgssc registered [ 7868.214050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7876.206298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7876.223812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7877.656058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7881.943492] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7887.847350] LustreError: 231697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7887.868351] LustreError: 231697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7891.020593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7891.419800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7895.759838] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7903.455421] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7909.096347] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7916.237526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7916.714270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7922.715848] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7922.871948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7930.962601] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7948.769135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7948.776536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7948.802518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7951.840263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7951.851460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7951.861689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7951.884764] Lustre: Skipped 1 previous similar message [ 7953.772045] Lustre: server umount lustre-OST0000 complete [ 7957.984325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7957.994311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7958.011220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7962.081983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7962.085748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7967.204666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7967.211040] Lustre: Skipped 2 previous similar messages [ 7972.319136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7972.493991] Lustre: server umount lustre-MDT0000 complete [ 7976.379923] LustreError: 233539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084670 with bad export cookie 12252299431099384310 [ 7976.385150] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7976.390790] LustreError: 233539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7976.787570] Lustre: server umount lustre-MDT0001 complete [ 7986.380925] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7989.011370] Key type lgssc unregistered [ 7989.241680] LNet: 235375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7989.249942] LNetError: 235375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7989.276488] LNet: Removed LNI 192.168.204.145@tcp [ 7990.158075] Key type .llcrypt unregistered [ 7990.162213] Key type ._llcrypt unregistered [ 8011.032250] Key type ._llcrypt registered [ 8011.037550] Key type .llcrypt registered [ 8011.150101] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8011.904361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8011.939996] alg: No test for adler32 (adler32-zlib) [ 8013.031284] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 8013.333447] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8015.024150] Key type lgssc registered [ 8016.139496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8288.621869] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8291.580628] Key type lgssc unregistered [ 8291.908691] LNet: 237356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8291.914517] LNetError: 237356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8291.927584] LNet: Removed LNI 192.168.204.145@tcp [ 8292.655574] Key type .llcrypt unregistered [ 8292.661558] Key type ._llcrypt unregistered [ 8320.866362] Key type ._llcrypt registered [ 8320.869290] Key type .llcrypt registered [ 8321.002789] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 8336.792943] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8337.917441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8338.069157] alg: No test for adler32 (adler32-zlib) [ 8339.175431] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 8339.448796] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8341.128148] Key type lgssc registered [ 8342.278408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8374.443726] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8043 sec ========== 12:57:46 (1785085066) [ 8376.317972] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:57:47 (1785085067) === [ 8379.848098] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:57:51 (1785085071) === [ 8426.110622] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8430.128692] Key type lgssc unregistered [ 8430.521622] LNet: 242576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8430.530350] LNetError: 242576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8430.591781] LNet: Removed LNI 192.168.204.145@tcp [ 8431.601485] Key type .llcrypt unregistered [ 8431.621057] Key type ._llcrypt unregistered