[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 437597456 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002305] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007544] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010127] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013085] *** VALIDATE selinux *** [ 0.022788] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027619] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029140] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] *** VALIDATE tmpfs *** [ 0.032159] *** VALIDATE proc *** [ 0.033243] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035434] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036226] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.043300] debug: unmapping init [mem 0xffffffffa2459000-0xffffffffa2460fff] [ 0.046123] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047737] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050015] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055344] rcu: Hierarchical SRCU implementation. [ 0.057573] smp: Bringing up secondary CPUs ... [ 0.058654] x86: Booting SMP configuration: [ 0.059030] .... node #0, CPUs: #1 #2 #3 [ 0.066084] smp: Brought up 1 node, 4 CPUs [ 0.068036] smpboot: Max logical packages: 1 [ 0.069013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148090] node 0 deferred pages initialised in 77ms [ 0.151134] devtmpfs: initialized [ 0.152226] x86/mm: Memory block size: 128MB [ 0.154818] gcov: version magic: 0x41383552 [ 0.156127] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158250] pinctrl core: initialized pinctrl subsystem [ 0.159214] [ 0.159587] ************************************************************* [ 0.160014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161010] ** ** [ 0.162014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163011] ** ** [ 0.164014] ** This means that this kernel is built to expose internal ** [ 0.165013] ** IOMMU data structures, which may compromise security on ** [ 0.166013] ** your system. ** [ 0.167020] ** ** [ 0.168014] ** If you see this message and you are not debugging the ** [ 0.169014] ** kernel, report this immediately to your vendor! ** [ 0.170013] ** ** [ 0.171014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172018] ************************************************************* [ 0.173665] NET: Registered protocol family 16 [ 0.174445] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.178050] cpuidle: using governor menu [ 0.179760] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182629] PCI: Using configuration type 1 for base access [ 0.185183] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194103] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197033] cryptd: max_cpu_qlen set to 1000 [ 0.199318] ACPI: Added _OSI(Module Device) [ 0.200014] ACPI: Added _OSI(Processor Device) [ 0.201018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206093] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209088] ACPI: Interpreter enabled [ 0.210058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211015] ACPI: Using IOAPIC for interrupt routing [ 0.212145] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.213384] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.224096] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.225032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.226019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.227147] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.229307] acpiphp: Slot [2] registered [ 0.230145] acpiphp: Slot [5] registered [ 0.231132] acpiphp: Slot [6] registered [ 0.232104] acpiphp: Slot [7] registered [ 0.233171] acpiphp: Slot [8] registered [ 0.234112] acpiphp: Slot [9] registered [ 0.235166] acpiphp: Slot [10] registered [ 0.236126] acpiphp: Slot [3] registered [ 0.237110] acpiphp: Slot [4] registered [ 0.238178] acpiphp: Slot [11] registered [ 0.239090] acpiphp: Slot [12] registered [ 0.240159] acpiphp: Slot [13] registered [ 0.241103] acpiphp: Slot [14] registered [ 0.242097] acpiphp: Slot [15] registered [ 0.243120] acpiphp: Slot [16] registered [ 0.244081] acpiphp: Slot [17] registered [ 0.245073] acpiphp: Slot [18] registered [ 0.246102] acpiphp: Slot [19] registered [ 0.247093] acpiphp: Slot [20] registered [ 0.248085] acpiphp: Slot [21] registered [ 0.249132] acpiphp: Slot [22] registered [ 0.250097] acpiphp: Slot [23] registered [ 0.251095] acpiphp: Slot [24] registered [ 0.252112] acpiphp: Slot [25] registered [ 0.253130] acpiphp: Slot [26] registered [ 0.254107] acpiphp: Slot [27] registered [ 0.255081] acpiphp: Slot [28] registered [ 0.256124] acpiphp: Slot [29] registered [ 0.257098] acpiphp: Slot [30] registered [ 0.258101] acpiphp: Slot [31] registered [ 0.259072] PCI host bridge to bus 0000:00 [ 0.260071] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.261018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.262022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.263024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.264025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.265025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.266157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.268082] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.270333] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.276016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.281036] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.282018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.283021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.284015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.285594] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.286925] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.287054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.288817] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.290866] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.296013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.299017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.302791] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.306019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.309022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.317017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.321000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.334017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.341017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.361020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.372062] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.383022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.393016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.414024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.427700] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.439017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.448017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.464018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.471859] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.477016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.482015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.495016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.503595] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.507015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.512014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.523018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.528658] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.530199] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.532327] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.534229] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.535119] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.538033] iommu: Default domain type: Passthrough [ 0.539333] SCSI subsystem initialized [ 0.540093] ACPI: bus type USB registered [ 0.542109] usbcore: registered new interface driver usbfs [ 0.544100] usbcore: registered new interface driver hub [ 0.546099] usbcore: registered new device driver usb [ 0.547127] pps_core: LinuxPPS API ver. 1 registered [ 0.548007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.550056] PTP clock support registered [ 0.551203] EDAC MC: Ver: 3.0.0 [ 0.553125] PCI: Using ACPI for IRQ routing [ 0.554620] NetLabel: Initializing [ 0.555010] NetLabel: domain hash size = 128 [ 0.555864] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.557076] NetLabel: unlabeled traffic allowed by default [ 0.558211] vgaarb: loaded [ 0.559254] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.559982] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.566006] clocksource: Switched to clocksource kvm-clock [ 0.685129] VFS: Disk quotas dquot_6.6.0 [ 0.686921] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.689669] *** VALIDATE ramfs *** [ 0.690985] *** VALIDATE hugetlbfs *** [ 0.692776] pnp: PnP ACPI init [ 0.696152] pnp: PnP ACPI: found 6 devices [ 0.712170] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.715839] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.718040] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.720401] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.723001] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.725645] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.728637] NET: Registered protocol family 2 [ 0.731176] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.736249] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.739850] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.745620] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.749711] TCP: Hash tables configured (established 65536 bind 65536) [ 0.752671] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.755388] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.758276] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.761358] NET: Registered protocol family 1 [ 0.763295] RPC: Registered named UNIX socket transport module. [ 0.764541] RPC: Registered udp transport module. [ 0.765974] RPC: Registered tcp transport module. [ 0.767199] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.768605] NET: Registered protocol family 44 [ 0.769569] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.771169] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.773361] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.774772] PCI: CLS 0 bytes, default 64 [ 0.775867] Unpacking initramfs... [ 2.091130] debug: unmapping init [mem 0xffff9211bcc54000-0xffff9211bffbffff] [ 2.094183] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.096139] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.098932] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.600708] Initialise system trusted keyrings [ 2.602500] Key type blacklist registered [ 2.604080] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.612293] zbud: loaded [ 2.614456] *** VALIDATE nfs *** [ 2.615296] *** VALIDATE nfs4 *** [ 2.616230] pstore: using deflate compression [ 2.618829] Platform Keyring initialized [ 2.693403] NET: Registered protocol family 38 [ 2.694527] Key type asymmetric registered [ 2.695734] Asymmetric key parser 'x509' registered [ 2.696797] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.698923] io scheduler mq-deadline registered [ 2.700130] io scheduler kyber registered [ 2.701296] io scheduler bfq registered [ 2.702435] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.704721] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.706531] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.708353] ACPI: Power Button [PWRF] [ 2.711956] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.717565] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.726914] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.731637] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.742093] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.767421] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.792988] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.796486] Non-volatile memory driver v1.3 [ 2.797299] Linux agpgart interface v0.103 [ 2.823505] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.825692] vda: detected capacity change from 0 to 74702848 [ 2.843250] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.845184] vdb: detected capacity change from 0 to 1073741824 [ 2.858429] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.860260] vdc: detected capacity change from 0 to 2621440000 [ 2.872075] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.874059] vdd: detected capacity change from 0 to 2621440000 [ 2.885147] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.887796] vde: detected capacity change from 0 to 4294967296 [ 2.905766] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.908657] vdf: detected capacity change from 0 to 4294967296 [ 2.920942] libphy: Fixed MDIO Bus: probed [ 2.927741] usbcore: registered new interface driver usbserial_generic [ 2.929322] usbserial: USB Serial support registered for generic [ 2.931129] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.934770] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.936086] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.938216] mousedev: PS/2 mouse device common for all mice [ 2.940484] rtc_cmos 00:05: RTC can wake from S4 [ 2.945484] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.948489] rtc_cmos 00:05: registered as rtc0 [ 2.950305] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.950955] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.950987] intel_pstate: CPU model not supported [ 2.952198] hid: raw HID events driver (C) Jiri Kosina [ 2.957187] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.958213] usbcore: registered new interface driver usbhid [ 2.958217] usbhid: USB HID core driver [ 2.959504] drop_monitor: Initializing network drop monitor service [ 2.969907] Initializing XFRM netlink socket [ 2.971416] NET: Registered protocol family 10 [ 2.974072] Segment Routing with IPv6 [ 2.975153] NET: Registered protocol family 17 [ 2.976810] mpls_gso: MPLS GSO support [ 2.982824] RAS: Correctable Errors collector initialized. [ 2.984335] AVX version of gcm_enc/dec engaged. [ 2.985425] AES CTR mode by8 optimization enabled [ 3.056842] sched_clock: Marking stable (3056809788, 0)->(4129802278, -1072992490) [ 3.059805] registered taskstats version 1 [ 3.061513] Loading compiled-in X.509 certificates [ 3.063884] zswap: loaded using pool lzo/zbud [ 3.084417] Key type big_key registered [ 3.097099] Key type encrypted registered [ 3.098153] ima: No TPM chip found, activating TPM-bypass! [ 3.099589] ima: Allocated hash algorithm: sha1 [ 3.100707] ima: No architecture policies found [ 3.102094] evm: Initialising EVM extended attributes: [ 3.103910] evm: security.selinux [ 3.105113] evm: security.ima [ 3.106202] evm: security.capability [ 3.107470] evm: HMAC attrs: 0x1 [ 3.109754] rtc_cmos 00:05: setting system clock to 2026-08-09 21:20:00 UTC (1786310400) [ 3.115509] debug: unmapping init [mem 0xffffffffa3403000-0xffffffffa35fffff] [ 3.118182] debug: unmapping init [mem 0xffffffffa2182000-0xffffffffa2458fff] [ 3.128154] Write protecting the kernel read-only data: 28672k [ 3.130964] debug: unmapping init [mem 0xffffffffa0803000-0xffffffffa09fffff] [ 3.133084] debug: unmapping init [mem 0xffffffffa1114000-0xffffffffa11fffff] [ 3.162763] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.170028] systemd[1]: Detected virtualization kvm. [ 3.171694] systemd[1]: Detected architecture x86-64. [ 3.173222] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.197774] systemd[1]: No hostname configured. [ 3.198907] systemd[1]: Set hostname to . [ 3.200759] random: systemd: uninitialized urandom read (16 bytes read) [ 3.202558] systemd[1]: Initializing machine ID from random generator. [ 3.248359] random: ln: uninitialized urandom read (6 bytes read) [ 3.330402] random: systemd: uninitialized urandom read (16 bytes read) [ 3.333153] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.337615] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.342694] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.965645] device-mapper: uevent: version 1.0.3 [ 3.967836] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.692120] virtio_net virtio0 ens2: renamed from eth0 [ 4.705138] random: fast init done [ 4.814236] scsi host0: ata_piix [ 4.833524] scsi host1: ata_piix [ 4.835965] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.838538] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.084504] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.612459] random: crng init done [ 9.613845] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.009903] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.188124] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.433513] SELinux: Disabled at runtime. [ 11.495658] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.502431] systemd[1]: Detected virtualization kvm. [ 11.504080] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.052614] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.056441] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.062296] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.066268] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.069287] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.077339] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.082441] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.165681] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.627989] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.004036] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.022507] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.196331] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.207994] EDAC sbridge: Ver: 1.1.2 [ 14.800254] Key type dns_resolver registered [ 15.134823] NFS: Registering the id_resolver key type [ 15.136862] Key type id_resolver registered [ 15.138340] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg229-server login: [ 41.394504] libcfs: loading out-of-tree module taints kernel. [ 41.481659] Key type ._llcrypt registered [ 41.484570] Key type .llcrypt registered [ 41.619199] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 59.341778] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 60.918485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.936795] alg: No test for adler32 (adler32-zlib) [ 62.453962] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 63.484483] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 65.248225] Key type lgssc registered [ 66.928493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.164066] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 117.088423] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 129.664547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 129.683901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 130.987455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 131.019271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 131.201788] Lustre: lustre-MDT0000: new disk, initializing [ 131.361878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 131.391615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 135.816083] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 149.094990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.222677] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 149.252483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 149.259619] Lustre: Skipped 1 previous similar message [ 149.374953] Lustre: lustre-MDT0001: new disk, initializing [ 149.494647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.553155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 149.574635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 154.922816] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.596685] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 168.131238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 168.468142] Lustre: lustre-OST0000: new disk, initializing [ 168.476228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 168.483347] Lustre: 8442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 168.573398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.255087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 172.266877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 172.353324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 175.061244] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.063689] hrtimer: interrupt took 4770566 ns [ 189.964738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.122691] Lustre: lustre-OST0001: new disk, initializing [ 190.126690] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 190.130768] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.201584] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.202544] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.196299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 197.211952] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 197.273549] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 209.157571] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 215.753603] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 224.064605] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing check_logdir /tmp/testlogs/ [ 230.271315] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing yml_node [ 235.091624] Lustre: DEBUG MARKER: Client: 2.17.56.55 [ 237.402798] Lustre: DEBUG MARKER: MDS: 2.17.56.55 [ 239.980699] Lustre: DEBUG MARKER: OSS: 2.17.56.55 [ 241.752253] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 9 17:23:58 EDT 2026 [ 258.656474] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 260.327987] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 277.473727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 277.503724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 277.514339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 279.010570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.011595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 279.035345] Lustre: Skipped 1 previous similar message [ 279.058119] Lustre: Skipped 3 previous similar messages [ 281.792571] Lustre: server umount lustre-MDT0000 complete [ 285.391475] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310682 with bad export cookie 2561411065166589261 [ 285.405099] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 289.248780] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 289.250036] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.272144] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 289.300368] Lustre: Skipped 1 previous similar message [ 289.305657] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 290.272900] LustreError: 8908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.308549] LustreError: 8908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 291.810261] Lustre: server umount lustre-MDT0001 complete [ 295.270440] Lustre: server umount lustre-OST0000 complete [ 298.919947] Lustre: server umount lustre-OST0001 complete [ 305.584104] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 314.257823] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 345.489659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.757991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 345.790308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 345.859924] Lustre: lustre-MDT0000: new disk, initializing [ 345.951953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 345.965663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 350.370271] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 359.863794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 360.018774] Lustre: 15067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 360.083245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 360.087235] Lustre: Skipped 1 previous similar message [ 360.198961] Lustre: lustre-MDT0001: new disk, initializing [ 360.328720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 360.366721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 360.374826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 364.698718] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 375.098156] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 381.031348] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 387.420674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 387.818925] Lustre: lustre-OST0000: new disk, initializing [ 387.824455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 387.829685] Lustre: 17038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 387.908178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.618132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 389.635819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 389.857803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 395.127302] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 405.307348] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 410.538829] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 410.768687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 416.693663] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 416.904638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 420.327933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 420.332729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.341802] Lustre: Skipped 1 previous similar message [ 420.347829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 420.353583] Lustre: Skipped 3 previous similar messages [ 425.958652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 425.968274] Lustre: Skipped 2 previous similar messages [ 433.119614] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 433.316612] Lustre: server umount lustre-OST0000 complete [ 441.316970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.326623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 441.331272] Lustre: Skipped 2 previous similar messages [ 441.346674] Lustre: Skipped 3 previous similar messages [ 444.268897] Lustre: server umount lustre-MDT0000 complete [ 446.437694] LustreError: 15077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.446762] LustreError: 15077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 448.349249] LustreError: 15056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310845 with bad export cookie 2561411065166590794 [ 448.356818] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 448.358927] LustreError: 15056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 448.713340] Lustre: server umount lustre-MDT0001 complete [ 451.887747] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:27:28 (1786310848) [ 458.562953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.267427] LustreError: 19304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 459.378380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.875248] LustreError: 19305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.058317] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 474.086493] LustreError: 19304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.101150] LustreError: 19304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 475.183894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.311763] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 488.639855] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 495.353798] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 503.792605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 512.339164] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 521.987980] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 544.229973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.244911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.246333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.250401] LustreError: Skipped 1 previous similar message [ 544.286835] Lustre: Skipped 1 previous similar message [ 546.075269] Lustre: server umount lustre-OST0000 complete [ 552.421331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 557.971293] Lustre: server umount lustre-MDT0000 complete [ 559.599053] LustreError: 19304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.623718] LustreError: 19304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 562.555853] LustreError: 19284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310959 with bad export cookie 2561411065166591676 [ 562.582764] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 562.835159] Lustre: server umount lustre-MDT0001 complete [ 574.695772] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 580.128891] Key type lgssc unregistered [ 580.696976] LNet: 22861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 580.712491] LNetError: 22861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 580.746979] LNet: Removed LNI 192.168.202.129@tcp [ 583.143171] Key type .llcrypt unregistered [ 583.146609] Key type ._llcrypt unregistered [ 609.105593] Key type ._llcrypt registered [ 609.107637] Key type .llcrypt registered [ 609.232620] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 610.496147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 610.524837] alg: No test for adler32 (adler32-zlib) [ 611.702805] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 611.998733] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 613.704956] Key type lgssc registered [ 615.102314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 624.276761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 624.322736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.780544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.878587] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 636.400034] LustreError: 24276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 636.438548] LustreError: 24276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 639.895974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.292847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 645.056528] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 653.036540] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 658.926321] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 665.150190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 665.697076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 670.843274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 672.082635] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 681.516966] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 699.127261] Lustre: server umount lustre-OST0000 complete [ 706.024433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.027509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.048463] Lustre: Skipped 1 previous similar message [ 706.072147] Lustre: Skipped 1 previous similar message [ 711.137188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 711.165531] Lustre: Skipped 1 previous similar message [ 716.257374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 716.270471] Lustre: Skipped 1 previous similar message [ 717.279185] Lustre: 23585:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786311098/real 1786311098] req@000000004d500d2c x1873082852603264/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786311114 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 717.282418] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 717.283790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.300719] Lustre: 23585:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 717.548890] Lustre: server umount lustre-MDT0000 complete [ 722.228872] LustreError: 24257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311119 with bad export cookie 1370393008621206533 [ 722.235296] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.242377] LustreError: 24257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 722.665798] Lustre: server umount lustre-MDT0001 complete [ 735.974838] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 739.499724] Key type lgssc unregistered [ 739.807856] LNet: 28258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.811527] LNetError: 28258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.828858] LNet: Removed LNI 192.168.202.129@tcp [ 741.060535] Key type .llcrypt unregistered [ 741.062212] Key type ._llcrypt unregistered [ 765.322195] Key type ._llcrypt registered [ 765.328322] Key type .llcrypt registered [ 765.511504] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 766.687890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 766.782406] alg: No test for adler32 (adler32-zlib) [ 767.916645] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 768.179650] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 769.863251] Key type lgssc registered [ 771.156292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 781.110655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 781.133611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 782.637395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 787.468214] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 792.052300] LustreError: 29671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 792.067205] LustreError: 29671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 797.155534] LustreError: 29672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 800.338490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.836235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 807.660148] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 817.098986] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 824.736538] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 839.568422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 840.156869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.595844] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 852.477492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 855.350773] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 866.790469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 866.801420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.820786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 867.815371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 867.834108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 867.849437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.198842] Lustre: server umount lustre-OST0000 complete [ 877.536472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 877.541485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.551441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 877.555713] Lustre: Skipped 1 previous similar message [ 881.164945] Lustre: server umount lustre-MDT0000 complete [ 883.172718] LustreError: 29667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 883.192155] LustreError: 29667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 884.742146] LustreError: 29653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311282 with bad export cookie 38326490610787388 [ 884.752340] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 885.166692] Lustre: server umount lustre-MDT0001 complete [ 897.587412] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 901.065263] Key type lgssc unregistered [ 901.442234] LNet: 33551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.449931] LNetError: 33551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.471549] LNet: Removed LNI 192.168.202.129@tcp [ 902.606926] Key type .llcrypt unregistered [ 902.609473] Key type ._llcrypt unregistered [ 930.253870] Key type ._llcrypt registered [ 930.256371] Key type .llcrypt registered [ 930.370829] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 931.642150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 931.895468] alg: No test for adler32 (adler32-zlib) [ 933.025577] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 933.307426] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 935.047380] Key type lgssc registered [ 936.391898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 944.423604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 944.439842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 945.795572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.311729] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 956.390089] LustreError: 34967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.402034] LustreError: 34967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 960.259488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.641683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 966.219884] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 975.929933] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 983.882818] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 992.984078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 993.281321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.771129] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1009.326313] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1015.871174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1026.016899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.028058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1026.041287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1026.045725] LustreError: Skipped 1 previous similar message [ 1026.053444] Lustre: Skipped 1 previous similar message [ 1030.420407] Lustre: server umount lustre-OST0000 complete [ 1036.258612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.275280] Lustre: Skipped 1 previous similar message [ 1036.279601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1041.274551] Lustre: server umount lustre-MDT0000 complete [ 1041.378315] LustreError: 34967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.399930] LustreError: 34967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1044.883722] LustreError: 34948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311442 with bad export cookie 13195706034834666812 [ 1044.887477] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1044.894238] LustreError: 34948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1045.277331] Lustre: server umount lustre-MDT0001 complete [ 1056.113230] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 1059.200681] Key type lgssc unregistered [ 1059.541894] LNet: 38527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.547056] LNetError: 38527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.587302] LNet: Removed LNI 192.168.202.129@tcp [ 1060.533604] Key type .llcrypt unregistered [ 1060.537291] Key type ._llcrypt unregistered [ 1088.968302] Key type ._llcrypt registered [ 1088.969716] Key type .llcrypt registered [ 1089.052581] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1090.366949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1090.483023] alg: No test for adler32 (adler32-zlib) [ 1091.779560] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1092.021968] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 1093.735340] Key type lgssc registered [ 1095.249467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1104.156448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1104.190142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1105.878909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.460325] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1116.130461] LustreError: 39941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.160625] LustreError: 39941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1120.589860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1121.001378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1125.182121] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1133.294980] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1141.265174] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1149.608763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1150.243278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.292524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1158.662959] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1166.310480] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1174.372670] Lustre: lustre-OST0000: Not available for connect from 192.168.202.29@tcp (stopping) [ 1177.062223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1177.071063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1177.092946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1177.094651] Lustre: Skipped 1 previous similar message [ 1177.114677] Lustre: Skipped 1 previous similar message [ 1179.505373] Lustre: lustre-OST0000: Not available for connect from 192.168.202.29@tcp (stopping) [ 1182.182883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1182.197849] Lustre: Skipped 1 previous similar message [ 1187.299294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1187.307109] Lustre: Skipped 1 previous similar message [ 1188.321652] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1188.838539] Lustre: server umount lustre-OST0000 complete [ 1212.901951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.904356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1212.917384] Lustre: Skipped 1 previous similar message [ 1212.934794] Lustre: Skipped 1 previous similar message [ 1217.032410] Lustre: server umount lustre-MDT0000 complete [ 1217.059225] LustreError: 39941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.681617] LustreError: 40676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311619 with bad export cookie 8876708025554990747 [ 1221.698850] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1221.699148] LustreError: 40676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1228.279303] Lustre: server umount lustre-MDT0001 complete [ 1238.218967] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 1241.818353] Key type lgssc unregistered [ 1242.175734] LNet: 43654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.185055] LNetError: 43654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1242.194478] LNet: Removed LNI 192.168.202.129@tcp [ 1243.255277] Key type .llcrypt unregistered [ 1243.258671] Key type ._llcrypt unregistered [ 1270.646716] Key type ._llcrypt registered [ 1270.650392] Key type .llcrypt registered [ 1270.823204] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1272.374304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1272.452465] alg: No test for adler32 (adler32-zlib) [ 1273.646458] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1273.952406] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 1275.711221] Key type lgssc registered [ 1276.962422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1285.945872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1285.980763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.644850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.461847] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1296.873940] LustreError: 45067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1296.896473] LustreError: 45067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1301.994204] LustreError: 45068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.762404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.156410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1308.247796] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1316.501816] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1323.134541] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1330.419755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1330.843216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1331.916843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1338.040331] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1346.648473] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1353.126938] Lustre: server umount lustre-MDT0000 complete [ 1353.574825] LustreError: 45062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.702899] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.724447] Lustre: Skipped 2 previous similar messages [ 1357.273122] LustreError: 45049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311754 with bad export cookie 13594751405998247787 [ 1357.280061] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1357.288099] LustreError: 45049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1358.667287] LustreError: 45064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.692460] LustreError: 45064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1358.714295] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.29@tcp (stopping) [ 1358.820274] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.848857] LustreError: 45062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.849905] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1362.872633] LustreError: 45062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1362.880429] Lustre: Skipped 1 previous similar message [ 1364.203496] Lustre: server umount lustre-MDT0001 complete [ 1386.362691] Lustre: server umount lustre-OST0000 complete [ 1406.401857] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 1409.779716] Key type lgssc unregistered [ 1410.167940] LNet: 48932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.185729] LNetError: 48932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.233690] LNet: Removed LNI 192.168.202.129@tcp [ 1411.414819] Key type .llcrypt unregistered [ 1411.417666] Key type ._llcrypt unregistered [ 1438.381305] Key type ._llcrypt registered [ 1438.384877] Key type .llcrypt registered [ 1438.582215] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1440.137478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1440.186594] alg: No test for adler32 (adler32-zlib) [ 1441.601488] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1442.287062] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 1444.119194] Key type lgssc registered [ 1445.574069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1454.017369] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1454.069993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1469.861206] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1474.975445] LustreError: 50320:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.129@tcp: failed processing log, type 4: rc = -110 [ 1501.663836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1508.572516] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1516.877054] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1681.396982] Lustre: server umount lustre-OST0000 complete [ 1699.924222] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 1703.104227] Key type lgssc unregistered [ 1703.551955] LNet: 51998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.560895] LNetError: 51998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.612876] LNet: Removed LNI 192.168.202.129@tcp [ 1705.548925] Key type .llcrypt unregistered [ 1705.551201] Key type ._llcrypt unregistered [ 1734.205126] Key type ._llcrypt registered [ 1734.207697] Key type .llcrypt registered [ 1734.316239] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1736.024956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1736.191158] alg: No test for adler32 (adler32-zlib) [ 1737.261129] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1737.478300] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 1739.143230] Key type lgssc registered [ 1740.795559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1750.303283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1750.321397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.745904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.228521] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1762.286759] LustreError: 53414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.312221] LustreError: 53414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1766.962524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1767.502927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1773.626270] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1782.681790] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1789.339536] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1796.354963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1796.818866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.922249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1804.440848] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1814.122399] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1818.604871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1818.610307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.621551] LustreError: Skipped 1 previous similar message [ 1818.627337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1818.647054] Lustre: Skipped 1 previous similar message [ 1818.689528] Lustre: Skipped 1 previous similar message [ 1822.641810] Lustre: server umount lustre-OST0000 complete [ 1828.832993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.833242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1828.880949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1828.886724] Lustre: Skipped 1 previous similar message [ 1833.159319] Lustre: server umount lustre-MDT0000 complete [ 1833.961670] LustreError: 55717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1833.995840] LustreError: 55717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1837.658052] LustreError: 53396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312235 with bad export cookie 4742417690701418292 [ 1837.669542] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1837.672153] LustreError: 53396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1838.265527] Lustre: server umount lustre-MDT0001 complete [ 1853.108527] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 1856.684832] Key type lgssc unregistered [ 1856.973041] LNet: 56948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.979606] LNetError: 56948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.990734] LNet: Removed LNI 192.168.202.129@tcp [ 1858.099823] Key type .llcrypt unregistered [ 1858.102246] Key type ._llcrypt unregistered [ 1884.994833] Key type ._llcrypt registered [ 1884.998261] Key type .llcrypt registered [ 1885.115491] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1886.507025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1886.535787] alg: No test for adler32 (adler32-zlib) [ 1887.737365] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1888.044235] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 1889.759166] Key type lgssc registered [ 1890.945409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1899.131763] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1899.149085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1914.783526] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1919.904136] LustreError: 58338:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.129@tcp: failed processing log, type 4: rc = -110 [ 1946.599713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1954.218850] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1962.937860] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1969.106134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.753238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.916717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1974.753216] LustreError: 58364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.777409] LustreError: 58364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1976.025869] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1979.881537] LustreError: 58363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.905910] LustreError: 59375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.928489] LustreError: 59375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1985.900321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.018524] LustreError: 59374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1986.037833] LustreError: 59374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1986.312915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.562618] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2000.666823] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2008.147351] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2010.598532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2010.614319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.631868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2012.144345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2012.146367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2012.147991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.187704] Lustre: Skipped 1 previous similar message [ 2016.500752] Lustre: server umount lustre-OST0000 complete [ 2041.777034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.791018] Lustre: Skipped 1 previous similar message [ 2041.792801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2041.805637] Lustre: Skipped 1 previous similar message [ 2044.443059] Lustre: server umount lustre-MDT0000 complete [ 2046.963922] LustreError: 59379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.000364] LustreError: 59379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2048.695324] LustreError: 58345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312446 with bad export cookie 3305944680413609597 [ 2048.707172] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2048.708979] LustreError: 58345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2063.327152] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2063.946516] Lustre: server umount lustre-MDT0001 complete [ 2079.950137] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2083.341734] Key type lgssc unregistered [ 2083.536036] LNet: 62085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2083.541252] LNetError: 62085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2083.555765] LNet: Removed LNI 192.168.202.129@tcp [ 2084.927934] Key type .llcrypt unregistered [ 2084.929809] Key type ._llcrypt unregistered [ 2113.639429] Key type ._llcrypt registered [ 2113.644683] Key type .llcrypt registered [ 2113.822529] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2115.238741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2115.275684] alg: No test for adler32 (adler32-zlib) [ 2116.512725] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2116.856928] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2118.567587] Key type lgssc registered [ 2120.077597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2128.287638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2128.313199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2129.897383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.116748] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2140.146972] LustreError: 63498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2140.171519] LustreError: 63498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2145.255379] LustreError: 63499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.422120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.040939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2154.851436] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2164.804262] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2171.804544] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2180.728880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2181.204728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.898965] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2196.819305] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2199.663939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2219.489712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2219.493809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.511109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2220.514591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2220.518667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2220.563772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2222.415338] Lustre: server umount lustre-OST0000 complete [ 2230.242078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2230.255465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.281412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2230.293219] Lustre: Skipped 1 previous similar message [ 2232.736109] Lustre: server umount lustre-MDT0000 complete [ 2235.873020] LustreError: 63494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.906104] LustreError: 63494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2236.935540] LustreError: 63478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312634 with bad export cookie 11378978276651653851 [ 2236.952539] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2237.435070] Lustre: server umount lustre-MDT0001 complete [ 2250.156555] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2253.129248] Key type lgssc unregistered [ 2253.476737] LNet: 67065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.493931] LNetError: 67065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.521503] LNet: Removed LNI 192.168.202.129@tcp [ 2254.215558] Key type .llcrypt unregistered [ 2254.217724] Key type ._llcrypt unregistered [ 2285.795084] Key type ._llcrypt registered [ 2285.798343] Key type .llcrypt registered [ 2285.921975] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2287.796656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2287.850574] alg: No test for adler32 (adler32-zlib) [ 2289.027419] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2289.472661] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2291.207219] Key type lgssc registered [ 2292.844644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2302.073700] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 17:58:18 (1786312698) [ 2308.439377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2308.475222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.042148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2315.051560] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2320.354804] LustreError: 68770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.376338] LustreError: 68770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2323.855883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.397935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2330.107105] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2338.420613] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2344.340134] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2350.755816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2351.222829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2355.361717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2359.045482] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2369.396300] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2375.809388] Lustre: Failing over lustre-MDT0000 [ 2375.977397] Lustre: server umount lustre-MDT0000 complete [ 2376.547690] LustreError: 68766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.267589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2380.269287] LustreError: 68770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.286279] Lustre: Skipped 2 previous similar messages [ 2380.334205] LustreError: 68770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2385.386807] LustreError: 68766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.408768] LustreError: 68766:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2386.342649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.522697] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2386.717278] Lustre: *** cfs_fail_loc=135, val=0*** [ 2386.764395] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.29@tcp (not set up) [ 2386.874169] LustreError: 71859:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2386.884254] LustreError: 71859:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.129@tcp: cfg command failed: rc = -2 [ 2386.893219] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2386.893219] [ 2386.906029] LustreError: MGC192.168.202.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2386.914940] LustreError: 71847:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2386.918842] LustreError: 71847:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2386.924450] LustreError: 71847:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2386.931727] LustreError: 71847:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2386.959893] Lustre: server umount lustre-MDT0000 complete [ 2386.967289] LustreError: 71847:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2390.499584] LustreError: 69499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.525536] LustreError: 69499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2394.992760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.083248] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2395.366235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.431642] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2397.016087] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2400.584885] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2400.760834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2400.808921] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2400.874133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2410.976422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2410.983414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.985159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2410.994910] Lustre: Skipped 1 previous similar message [ 2411.006321] Lustre: Skipped 1 previous similar message [ 2415.926736] Lustre: server umount lustre-OST0000 complete [ 2421.219906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.238426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.245958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2421.254837] Lustre: Skipped 1 previous similar message [ 2421.304460] Lustre: Skipped 1 previous similar message [ 2426.337482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2426.346389] Lustre: Skipped 1 previous similar message [ 2426.720823] Lustre: server umount lustre-MDT0000 complete [ 2430.828725] LustreError: 68752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312828 with bad export cookie 7753018817177741281 [ 2430.834897] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.841957] LustreError: 68752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2431.328293] Lustre: server umount lustre-MDT0001 complete [ 2445.038574] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2448.413096] Key type lgssc unregistered [ 2448.667744] LNet: 73641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2448.674221] LNetError: 73641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2448.709917] LNet: Removed LNI 192.168.202.129@tcp [ 2449.398677] Key type .llcrypt unregistered [ 2449.400129] Key type ._llcrypt unregistered [ 2475.491485] Key type ._llcrypt registered [ 2475.493622] Key type .llcrypt registered [ 2475.626487] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2477.003479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2477.299128] alg: No test for adler32 (adler32-zlib) [ 2478.478527] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2478.754720] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2480.519187] Key type lgssc registered [ 2481.812919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2490.334241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2490.354433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.951862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.125482] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2502.131583] LustreError: 75072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.158933] LustreError: 75072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2506.605821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.040800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2512.308601] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2520.796419] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2527.546871] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2534.983221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2535.408225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.467056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2542.450347] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2550.674492] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2556.385616] Lustre: Failing over lustre-MDT0000 [ 2556.663417] Lustre: server umount lustre-MDT0000 complete [ 2556.901082] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.905696] LustreError: 75072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.914695] Lustre: Skipped 1 previous similar message [ 2562.018067] LustreError: 75071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.041702] LustreError: 75071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2565.687882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.834400] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2565.998385] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2566.201561] LustreError: 78159:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2566.224194] LustreError: 78159:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.129@tcp: cfg command failed: rc = -9 [ 2566.239278] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2566.239278] [ 2566.249854] LustreError: MGC192.168.202.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2566.269664] LustreError: 78147:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2566.284901] LustreError: 78147:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2566.295786] LustreError: 78147:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2566.303811] LustreError: 78147:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2566.322833] Lustre: server umount lustre-MDT0000 complete [ 2566.324856] LustreError: 78147:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2567.141670] LustreError: 75067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.151652] LustreError: 75067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2572.266934] LustreError: 75071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.283806] LustreError: 75071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2574.398607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.525804] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2574.879382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.960415] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2579.278848] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2579.946958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2580.003116] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2580.053392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2580.252508] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2590.179575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.198964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2590.210359] Lustre: Skipped 2 previous similar messages [ 2594.666875] Lustre: server umount lustre-OST0000 complete [ 2600.426953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.428591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2600.447162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2600.469212] Lustre: Skipped 2 previous similar messages [ 2605.537598] LustreError: 76744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.562268] LustreError: 76744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2605.642361] Lustre: server umount lustre-MDT0000 complete [ 2610.019486] LustreError: 75053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313007 with bad export cookie 13306704798242905612 [ 2610.023103] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2610.045554] LustreError: 75053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2610.458497] Lustre: server umount lustre-MDT0001 complete [ 2621.981303] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2625.318896] Key type lgssc unregistered [ 2625.776600] LNet: 79944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2625.792253] LNetError: 79944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2625.831204] LNet: Removed LNI 192.168.202.129@tcp [ 2626.839633] Key type .llcrypt unregistered [ 2626.840850] Key type ._llcrypt unregistered [ 2642.282769] Key type ._llcrypt registered [ 2642.286951] Key type .llcrypt registered [ 2642.461748] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2642.820462] Key type .llcrypt unregistered [ 2642.822572] Key type ._llcrypt unregistered [ 2667.392650] Key type ._llcrypt registered [ 2667.394026] Key type .llcrypt registered [ 2667.559111] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2669.071429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2669.106252] alg: No test for adler32 (adler32-zlib) [ 2670.154693] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2670.484686] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2672.215347] Key type lgssc registered [ 2673.610227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2681.402974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2681.436362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.078371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.335608] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2693.615455] LustreError: 81851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2693.639090] LustreError: 81851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2696.598524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2697.121639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2701.923909] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2709.765572] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2715.835653] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2723.630734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2724.045606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2729.212122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2731.654903] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2739.680994] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2755.041085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2755.052434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.076074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2758.113977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2758.122404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.128258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.413677] Lustre: server umount lustre-OST0000 complete [ 2768.352972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.356717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2768.374498] Lustre: Skipped 1 previous similar message [ 2768.380605] Lustre: Skipped 1 previous similar message [ 2771.358734] Lustre: server umount lustre-MDT0000 complete [ 2773.479610] LustreError: 82578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.499164] LustreError: 82578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2775.132410] LustreError: 81833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313172 with bad export cookie 2345311833064479805 [ 2775.150157] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2775.518782] Lustre: server umount lustre-MDT0001 complete [ 2786.707496] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2789.602322] Key type lgssc unregistered [ 2789.931661] LNet: 85409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2789.942521] LNetError: 85409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2789.959584] LNet: Removed LNI 192.168.202.129@tcp [ 2790.898115] Key type .llcrypt unregistered [ 2790.902252] Key type ._llcrypt unregistered [ 2816.247095] Key type ._llcrypt registered [ 2816.251672] Key type .llcrypt registered [ 2816.424316] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2818.020503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2818.082513] alg: No test for adler32 (adler32-zlib) [ 2819.312511] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2819.524103] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2821.167242] Key type lgssc registered [ 2822.354675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2830.739981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2830.778749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.303943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.321363] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2842.593847] LustreError: 86822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.609478] LustreError: 86822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2846.046027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.342326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2850.812951] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2858.119076] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2864.143228] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2870.968528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2871.224239] LustreError: 88642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.469271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.945022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2878.482814] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2886.954911] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2895.663276] Lustre: server umount lustre-OST0000 complete [ 2902.496738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2902.508245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.528358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2903.526931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2903.528190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.643893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2908.659488] Lustre: Skipped 2 previous similar messages [ 2909.938723] Lustre: server umount lustre-MDT0000 complete [ 2912.735496] Lustre: 86136:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786313294/real 1786313294] req@000000009bb19884 x1873085166807040/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786313310 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2912.747804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2912.761333] LustreError: 88255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.795443] LustreError: 88255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2913.823258] Lustre: 86136:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786313295/real 1786313295] req@00000000dadea15f x1873085166807936/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786313311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2914.104027] LustreError: 86805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313311 with bad export cookie 8254584967252266327 [ 2914.113725] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2914.116900] LustreError: 86805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2914.518626] Lustre: server umount lustre-MDT0001 complete [ 2925.964572] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2930.788496] Key type lgssc unregistered [ 2931.285781] LNet: 90358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2931.308105] LNetError: 90358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2932.325634] LNet: Removed LNI 192.168.202.129@tcp [ 2933.613958] Key type .llcrypt unregistered [ 2933.623972] Key type ._llcrypt unregistered [ 2959.516699] Key type ._llcrypt registered [ 2959.519659] Key type .llcrypt registered [ 2959.630986] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2960.796124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2960.883511] alg: No test for adler32 (adler32-zlib) [ 2962.077425] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2962.388028] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2964.079148] Key type lgssc registered [ 2965.272869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2974.763855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2974.785824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.422412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.479621] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2985.955320] LustreError: 91770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.988656] LustreError: 91770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2991.072523] LustreError: 91771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.312438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.766693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2998.144688] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3008.095628] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3014.615535] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3020.922812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3021.256082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3028.637798] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3037.144312] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3045.875537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3060.209243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3060.222206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.235966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3061.218977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.223193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3064.936831] Lustre: server umount lustre-OST0000 complete [ 3069.421153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3069.425282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.462689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3069.467965] Lustre: Skipped 1 previous similar message [ 3071.458779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3071.471594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3071.478028] Lustre: Skipped 1 previous similar message [ 3075.588399] Lustre: server umount lustre-MDT0000 complete [ 3076.577582] LustreError: 91770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3076.594651] LustreError: 91770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3078.991589] LustreError: 91753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313476 with bad export cookie 10796441665356198738 [ 3079.001310] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3079.001982] LustreError: 91753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3079.256489] Lustre: server umount lustre-MDT0001 complete [ 3089.976722] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 3093.116930] Key type lgssc unregistered [ 3093.444825] LNet: 95332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3093.461071] LNetError: 95332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3093.480885] LNet: Removed LNI 192.168.202.129@tcp [ 3094.637267] Key type .llcrypt unregistered [ 3094.642252] Key type ._llcrypt unregistered [ 3122.375156] Key type ._llcrypt registered [ 3122.382126] Key type .llcrypt registered [ 3122.544482] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 3123.744412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3123.812204] alg: No test for adler32 (adler32-zlib) [ 3124.893359] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 3125.077039] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 3126.743438] Key type lgssc registered [ 3128.039335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3137.180395] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3137.194974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3152.735455] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3157.858655] LustreError: 96719:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.129@tcp: failed processing log, type 4: rc = -110 [ 3184.607409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.026840] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3198.708402] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3204.031456] Lustre: server umount lustre-OST0000 complete [ 3213.627611] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:13:29 (1786313609) [ 3219.921428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.400241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.026512] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3225.572084] LustreError: 98358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.581230] LustreError: 98358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3229.673837] LustreError: 98359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.995828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.680846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3238.711762] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3248.627482] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3255.913917] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3262.837136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3263.270576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.353452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3270.385325] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3280.485954] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3294.177429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.179259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.198655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3294.213154] Lustre: Skipped 1 previous similar message [ 3299.314744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3299.329989] Lustre: Skipped 1 previous similar message [ 3299.764329] Lustre: server umount lustre-OST0000 complete [ 3304.417551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.421432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.429960] Lustre: Skipped 2 previous similar messages [ 3304.435081] Lustre: Skipped 1 previous similar message [ 3309.539818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.548274] Lustre: Skipped 1 previous similar message [ 3314.658399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3314.666106] Lustre: Skipped 1 previous similar message [ 3318.239213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3318.460857] Lustre: server umount lustre-MDT0000 complete [ 3319.777104] LustreError: 98353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.801975] LustreError: 98353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3323.053677] LustreError: 99094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313720 with bad export cookie 15160230819251678866 [ 3323.058471] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3323.072078] LustreError: 99094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3323.677105] Lustre: server umount lustre-MDT0001 complete [ 3335.484346] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 3338.473583] Key type lgssc unregistered [ 3338.693974] LNet: 102054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3338.703456] LNetError: 102054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3338.726798] LNet: Removed LNI 192.168.202.129@tcp [ 3339.626290] Key type .llcrypt unregistered [ 3339.628314] Key type ._llcrypt unregistered [ 3382.526158] Key type ._llcrypt registered [ 3382.533118] Key type .llcrypt registered [ 3382.643456] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 3383.664716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3383.678620] alg: No test for adler32 (adler32-zlib) [ 3384.943183] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 3385.332122] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 3387.061738] Key type lgssc registered [ 3388.392062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3397.827649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3397.873736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.477284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.877700] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3408.866823] LustreError: 104242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.901747] LustreError: 104242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3413.953648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3413.988075] LustreError: 104243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.338736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3419.155941] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3427.400350] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3433.980685] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3441.805473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3442.287159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3444.340379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3448.509895] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3456.867676] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3469.285563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.287485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3469.301959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.309843] Lustre: Skipped 1 previous similar message [ 3471.989310] Lustre: server umount lustre-OST0000 complete [ 3479.521380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.533192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3479.543950] Lustre: Skipped 1 previous similar message [ 3479.554656] Lustre: Skipped 1 previous similar message [ 3484.640907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3484.653150] Lustre: Skipped 2 previous similar messages [ 3488.740218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3488.745450] Lustre: Skipped 1 previous similar message [ 3491.295149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3491.611775] Lustre: server umount lustre-MDT0000 complete [ 3493.861898] LustreError: 105711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3493.881627] LustreError: 105711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3495.550788] LustreError: 104222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313892 with bad export cookie 7325005419532881730 [ 3495.575796] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3495.974968] Lustre: server umount lustre-MDT0001 complete [ 3508.602774] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 3512.055932] Key type lgssc unregistered [ 3512.507581] LNet: 107779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3512.527633] LNetError: 107779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3513.573886] LNet: Removed LNI 192.168.202.129@tcp [ 3514.853089] Key type .llcrypt unregistered [ 3514.856094] Key type ._llcrypt unregistered [ 3536.366404] Key type ._llcrypt registered [ 3536.368290] Key type .llcrypt registered [ 3536.475675] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 3538.126686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3538.182590] alg: No test for adler32 (adler32-zlib) [ 3539.288163] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 3539.652988] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 3541.428649] Key type lgssc registered [ 3543.086308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3552.340210] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3552.350798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3567.904200] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3573.026115] LustreError: 109021:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.129@tcp: failed processing log, type 4: rc = -110 [ 3599.775256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3606.906139] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3615.315085] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3622.680216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3622.957684] LustreError: MGC192.168.202.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3622.968786] LustreError: 110045:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3622.976800] LustreError: 110045:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3622.984814] LustreError: 110045:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3623.141763] Lustre: server umount lustre-MDT0000 complete [ 3623.150532] LustreError: 110045:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3642.847448] Lustre: 108359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786314024/real 1786314024] req@00000000e892dda8 x1873085922807168/t0(0) o400->MGC192.168.202.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786314040 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3642.867481] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3642.877719] LustreError: 109033:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3663.474481] Lustre: server umount lustre-OST0000 complete [ 3673.504598] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 3684.176606] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 3721.982881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.308480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3722.346257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3722.458074] Lustre: lustre-MDT0000: new disk, initializing [ 3722.555926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.570706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3726.994781] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3737.316679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3737.387628] Lustre: 112842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3737.420401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3737.422446] Lustre: Skipped 1 previous similar message [ 3737.465179] Lustre: lustre-MDT0001: new disk, initializing [ 3737.545164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3737.580828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3737.586364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3742.401847] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3752.042810] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3758.121642] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3765.100732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3765.435741] Lustre: lustre-OST0000: new disk, initializing [ 3765.444784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3765.462237] Lustre: 114812:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3765.608215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3772.756385] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3774.588173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3774.592572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3774.740043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3783.764856] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3788.877364] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3789.087905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3793.601179] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3793.827343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3800.551303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3800.551934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3800.554070] LustreError: Skipped 1 previous similar message [ 3800.555199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3800.577022] Lustre: Skipped 1 previous similar message [ 3805.666871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3805.681721] Lustre: Skipped 2 previous similar messages [ 3810.273911] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3810.746885] Lustre: server umount lustre-OST0000 complete [ 3819.488444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3819.497360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3819.510298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3821.025981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3826.144931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3826.147912] Lustre: Skipped 2 previous similar messages [ 3831.265187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3831.266634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3831.284180] Lustre: Skipped 2 previous similar messages [ 3831.502107] Lustre: server umount lustre-MDT0000 complete [ 3835.769108] LustreError: 112837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314233 with bad export cookie 8624401146022946921 [ 3835.791171] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.212943] Lustre: server umount lustre-MDT0001 complete [ 3846.148442] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:24:02 (1786314242) [ 3849.963187] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3865.691311] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 3876.882636] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 3914.758070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.976946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3914.998102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3915.047921] Lustre: lustre-MDT0000: new disk, initializing [ 3915.152774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3915.187760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3920.443770] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3931.084280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.211547] Lustre: 119718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3931.262495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3931.265730] Lustre: Skipped 1 previous similar message [ 3931.336252] Lustre: lustre-MDT0001: new disk, initializing [ 3931.484612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3931.518177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3931.530736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3936.192735] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3946.060353] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3951.666915] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3958.327034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3958.601585] Lustre: lustre-OST0000: new disk, initializing [ 3958.605268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3958.613380] Lustre: 121686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3958.704304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.343991] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3968.099986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3968.117671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3968.230246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3974.603729] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3979.948381] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3980.156477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3986.789277] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3987.193429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3994.081995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3994.086039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.096535] LustreError: Skipped 1 previous similar message [ 3994.110560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3994.130355] Lustre: Skipped 1 previous similar message [ 4004.319202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4004.722873] Lustre: server umount lustre-OST0000 complete [ 4008.416991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4008.419925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.443290] Lustre: Skipped 1 previous similar message [ 4014.633423] Lustre: server umount lustre-MDT0000 complete [ 4019.150657] LustreError: 119709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314416 with bad export cookie 8624401146022947803 [ 4019.171403] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4019.178086] LustreError: 119709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4019.935954] Lustre: server umount lustre-MDT0001 complete [ 4028.993786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4029.802357] LustreError: 123906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.828955] LustreError: 123906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4029.935405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4034.484210] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4035.043195] LustreError: 123907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4040.171303] LustreError: 123906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.101512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4047.592820] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4054.715825] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4060.931055] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4069.128548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4074.776284] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4081.288100] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4091.754581] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4098.533760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.542232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4098.560062] Lustre: Skipped 2 previous similar messages [ 4098.581055] Lustre: Skipped 8 previous similar messages [ 4102.662967] Lustre: server umount lustre-OST0000 complete [ 4112.987624] Lustre: server umount lustre-MDT0000 complete [ 4113.889374] LustreError: 125505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.908412] LustreError: 125505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4116.970200] LustreError: 123887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314514 with bad export cookie 8624401146022948678 [ 4116.975735] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4116.982289] LustreError: 123887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4117.630303] Lustre: server umount lustre-MDT0001 complete [ 4129.101424] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 4132.669758] Key type lgssc unregistered [ 4133.064792] LNet: 127539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4133.084428] LNetError: 127539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4133.116066] LNet: Removed LNI 192.168.202.129@tcp [ 4134.136329] Key type .llcrypt unregistered [ 4134.140106] Key type ._llcrypt unregistered [ 4152.644784] Key type ._llcrypt registered [ 4152.650654] Key type .llcrypt registered [ 4152.755703] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 4169.235651] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4171.295595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4171.364541] alg: No test for adler32 (adler32-zlib) [ 4172.634512] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 4172.966804] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 4174.671194] Key type lgssc registered [ 4176.481140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4214.662191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4214.670310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4215.821611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4215.858380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4215.900415] Lustre: lustre-MDT0000: new disk, initializing [ 4215.961936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.971842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4219.813031] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4232.839568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.086393] Lustre: 130696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4233.211335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4233.222571] Lustre: Skipped 1 previous similar message [ 4233.379342] Lustre: lustre-MDT0001: new disk, initializing [ 4233.542111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4233.624086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4233.648419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4239.583640] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4248.595495] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4254.215584] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4260.162714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4260.403920] Lustre: lustre-OST0000: new disk, initializing [ 4260.405712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4260.408369] Lustre: 132665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4260.477734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4266.952520] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4270.154465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4270.172137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4270.319414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4277.390287] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4282.531689] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4282.848398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4287.983176] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4288.277673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4291.043749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.049760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4291.056146] Lustre: Skipped 1 previous similar message [ 4296.173507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4296.181087] Lustre: Skipped 2 previous similar messages [ 4301.282056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4301.288356] Lustre: Skipped 1 previous similar message [ 4304.351195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4304.641167] Lustre: server umount lustre-OST0000 complete [ 4310.497428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4310.515332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.531177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4311.527231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.517397] Lustre: server umount lustre-MDT0000 complete [ 4318.292273] LustreError: 131502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314715 with bad export cookie 3111889262065919558 [ 4318.294914] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4318.298073] LustreError: 131502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4318.571275] Lustre: server umount lustre-MDT0001 complete [ 4326.908535] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:32:03 (1786314723) [ 4333.081867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4333.446149] LustreError: 135082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4333.461339] LustreError: 135082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4333.496780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4338.291652] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4338.663775] LustreError: 135083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4342.761693] LustreError: 135082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4345.997658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4346.353398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4350.905592] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4358.491637] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4365.526757] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4368.479032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.484350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4368.502487] Lustre: Skipped 1 previous similar message [ 4368.517597] Lustre: Skipped 3 previous similar messages [ 4374.754294] Lustre: server umount lustre-MDT0000 complete [ 4377.063779] LustreError: 135078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.088110] LustreError: 135078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4378.954371] LustreError: 135063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314776 with bad export cookie 3111889262065920433 [ 4378.973039] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.980076] LustreError: 135063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4379.679060] Lustre: server umount lustre-MDT0001 complete [ 4387.998190] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:33:03 (1786314783) [ 4395.582073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4411.231583] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4416.351452] LustreError: 137499:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.129@tcp: failed processing log, type 4: rc = -110 [ 4442.143268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4448.668438] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4456.671061] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4458.795501] Lustre: server umount lustre-OST0000 complete [ 4467.952468] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:34:24 (1786314864) [ 4475.180732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4475.780553] LustreError: 138944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.964655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.287368] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4486.126565] LustreError: 138945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.174223] LustreError: 138945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4492.317827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4499.254765] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4507.978273] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4513.046279] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4519.760753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4520.054629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4520.069896] Lustre: Skipped 1 previous similar message [ 4527.100987] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4535.841442] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4549.090439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4549.095073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.099101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4549.105500] Lustre: Skipped 2 previous similar messages [ 4549.120443] Lustre: Skipped 4 previous similar messages [ 4551.401615] Lustre: server umount lustre-MDT0000 complete [ 4554.216795] LustreError: 138944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4554.230853] LustreError: 138944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4555.437663] LustreError: 138925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314952 with bad export cookie 3111889262065921105 [ 4555.445924] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4555.453862] LustreError: 138925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4555.763913] Lustre: server umount lustre-MDT0001 complete [ 4570.740721] Lustre: server umount lustre-OST0000 complete [ 4579.835191] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:36:16 (1786314976) [ 4587.281183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4587.863232] LustreError: 142600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4587.969402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4592.700968] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4601.228842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4606.320734] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4613.738465] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4618.999452] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4625.607382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4632.420588] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4633.214701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4640.161790] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4645.117717] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4645.385823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4650.592674] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4650.893456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4654.063364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4654.071749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4654.084741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4667.360467] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4667.594639] Lustre: server umount lustre-OST0000 complete [ 4673.504537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4673.507763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4673.540186] Lustre: Skipped 1 previous similar message [ 4677.708093] Lustre: server umount lustre-MDT0000 complete [ 4679.648908] LustreError: 143612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4679.667598] LustreError: 143612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4681.390512] LustreError: 142580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315078 with bad export cookie 3111889262065922176 [ 4681.400193] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4681.401185] LustreError: 142580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4681.619372] Lustre: server umount lustre-MDT0001 complete [ 4689.922906] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:38:06 (1786315086) [ 4697.427822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4713.056936] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4718.175525] LustreError: 146759:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.129@tcp: failed processing log, type 4: rc = -110 [ 4743.904041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4743.914290] Lustre: Skipped 2 previous similar messages [ 4750.664176] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4757.999171] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4764.940382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4765.427624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4769.482206] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4777.524957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4782.332182] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4789.956341] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4796.503585] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4801.801153] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4802.123138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4807.421895] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4807.798888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4813.797967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4813.805636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.818134] Lustre: Skipped 1 previous similar message [ 4813.864285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4813.884215] Lustre: Skipped 8 previous similar messages [ 4816.709821] Lustre: server umount lustre-MDT0000 complete [ 4819.939780] LustreError: 147796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4819.950344] LustreError: 147796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4820.892726] LustreError: 146767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315218 with bad export cookie 3111889262065923198 [ 4820.907374] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4820.912221] LustreError: 146767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4821.340816] Lustre: server umount lustre-MDT0001 complete [ 4835.146370] Lustre: server umount lustre-OST0000 complete [ 4845.059190] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:40:41 (1786315241) [ 4853.540556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4869.089129] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4874.208763] LustreError: 150956:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.129@tcp: failed processing log, type 4: rc = -110 [ 4907.728647] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4917.111444] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4925.559122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4926.339063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4931.608981] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4941.676043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4947.558362] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4956.484728] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4963.006404] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4970.665930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4970.902450] Lustre: lustre-OST0001: new disk, initializing [ 4970.908348] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4970.914185] Lustre: 153830:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4978.176431] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4979.305318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4979.327765] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4979.429140] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4988.278494] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4994.251039] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4994.665776] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5000.064094] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5000.390362] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5003.232821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5003.246968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5003.258281] Lustre: Skipped 2 previous similar messages [ 5003.265727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5003.268458] Lustre: Skipped 3 previous similar messages [ 5008.768660] Lustre: server umount lustre-OST0000 complete [ 5028.319169] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5028.487220] Lustre: server umount lustre-OST0001 complete [ 5034.466375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5034.480544] LustreError: Skipped 3 previous similar messages [ 5039.569743] Lustre: server umount lustre-MDT0000 complete [ 5042.842734] LustreError: 150963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315440 with bad export cookie 3111889262065924150 [ 5042.847060] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5042.854826] LustreError: 150963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5043.327627] Lustre: server umount lustre-MDT0001 complete [ 5074.607089] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:44:30 (1786315470) [ 5076.502875] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5078.193274] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5080.563678] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:44:36 (1786315476) [ 5088.611159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.192995] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5089.283143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5089.606218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5089.610119] Lustre: Skipped 6 previous similar messages [ 5094.961677] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5105.306195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5105.477138] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5105.534537] Lustre: 157395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5111.439986] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5120.712422] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5127.256950] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5134.638434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5134.934232] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5142.572041] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5145.316656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5151.078430] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5156.814680] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5157.073152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5162.225339] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5162.540587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5166.073508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5166.088169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5166.100809] Lustre: Skipped 5 previous similar messages [ 5166.109275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5166.114732] Lustre: Skipped 14 previous similar messages [ 5170.731888] Lustre: server umount lustre-OST0000 complete [ 5196.156745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5204.741549] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5206.011905] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.038105] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5206.047493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.083841] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5219.414516] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5219.756805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5225.206507] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5225.452570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5231.854816] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8dcb461fe000.ost_server_uuid 50 [ 5234.153910] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8dcb461fe000.ost_server_uuid in FULL state after 0 sec [ 5234.362561] LustreError: 157402:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.29@tcp) returned error from blocking AST (req@000000005814d23e x1873086586830336 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000049f5b970/0x2b2fa6cbddcac190 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.29@tcp remote: 0xa750f56891ed5eae expref: 5 pid: 157402 timeout: 5334 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5234.378962] LustreError: lustre-MDT0000: A client on nid 192.168.202.29@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5234.384271] LustreError: 157390:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.29@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000049f5b970/0x2b2fa6cbddcac190 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.29@tcp remote: 0xa750f56891ed5eae expref: 6 pid: 157402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5239.784258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5239.789028] LustreError: Skipped 1 previous similar message [ 5245.395749] Lustre: server umount lustre-OST0000 complete [ 5264.352788] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5264.709857] Lustre: server umount lustre-MDT0000 complete [ 5267.426472] LustreError: 157405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5267.444142] LustreError: 157405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 5268.671809] LustreError: 157386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315666 with bad export cookie 3111889262065925326 [ 5268.673304] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5268.686554] LustreError: 157386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5275.136219] Lustre: server umount lustre-MDT0001 complete [ 5285.830762] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5289.567681] Key type lgssc unregistered [ 5289.925705] LNet: 163517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5289.931486] LNetError: 163517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5289.946724] LNet: Removed LNI 192.168.202.129@tcp [ 5291.033742] Key type .llcrypt unregistered [ 5291.035644] Key type ._llcrypt unregistered [ 5317.929162] Key type ._llcrypt registered [ 5317.931234] Key type .llcrypt registered [ 5318.125444] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5319.297213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5319.328145] alg: No test for adler32 (adler32-zlib) [ 5320.731694] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 5321.003113] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 5322.799233] Key type lgssc registered [ 5324.149362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5332.243274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5332.262356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.709541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.849763] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5344.225877] LustreError: 164949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5344.242677] LustreError: 164949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5347.621375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.110500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5353.545978] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5362.774453] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5369.786509] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5377.516713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5377.897901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5378.949157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5385.021298] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5393.157692] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5398.949252] Lustre: Failing over lustre-MDT0000 [ 5399.520191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5399.527991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.541883] LustreError: 165676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5399.607959] Lustre: server umount lustre-MDT0000 complete [ 5403.622816] LustreError: 164948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5403.630780] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5403.642041] LustreError: 164948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5408.738960] LustreError: 164945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.776809] LustreError: 164945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5411.851178] LustreError: 164929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.29@tcp arrived at 1786315809 with bad export cookie 14570852454080197308 [ 5411.887568] LustreError: 164929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5413.858076] LustreError: 167644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.884681] LustreError: 167644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5419.299088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5419.512838] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5419.866552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5419.957289] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5425.124781] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5425.136591] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5425.561370] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5428.473973] Lustre: lustre-MDT0000: Denying connection for new client 0cfb01c5-57de-4f25-8757-925d757aae2a (at 192.168.202.29@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5439.469686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5439.482556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5439.504530] Lustre: Skipped 1 previous similar message [ 5439.515295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5440.481799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5440.494798] Lustre: Skipped 1 previous similar message [ 5443.486422] Lustre: server umount lustre-OST0000 complete [ 5447.519603] LustreError: 168688:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5447.532193] Lustre: 168037:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5447.542227] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5447.566877] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5447.573730] LustreError: 168037:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000074a9aa99 x1873087790364928/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5447.603383] LustreError: 168037:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5447.615669] LustreError: 168037:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5447.623900] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5447.964366] Lustre: server umount lustre-MDT0000 complete [ 5450.723341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.731414] Lustre: Skipped 1 previous similar message [ 5450.735600] LustreError: 164945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5450.745211] LustreError: 164945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5451.988206] LustreError: 164931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315849 with bad export cookie 14570852454080197406 [ 5451.999489] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5452.010948] LustreError: 164931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5454.422807] Lustre: server umount lustre-MDT0001 complete [ 5465.526776] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5468.719391] Key type lgssc unregistered [ 5469.001810] LNet: 169443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5469.011348] LNetError: 169443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5469.032376] LNet: Removed LNI 192.168.202.129@tcp [ 5470.107993] Key type .llcrypt unregistered [ 5470.112358] Key type ._llcrypt unregistered [ 5497.237600] Key type ._llcrypt registered [ 5497.240636] Key type .llcrypt registered [ 5497.375730] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5498.449275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5498.726057] alg: No test for adler32 (adler32-zlib) [ 5499.869904] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 5500.226750] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 5501.935176] Key type lgssc registered [ 5503.222544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5512.151514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5512.194988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5513.909894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5518.136730] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5524.449237] LustreError: 170857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5524.461664] LustreError: 170857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5527.241418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.663588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5533.306558] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5541.328494] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5547.663496] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5555.307210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5555.791545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5558.831212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5562.343526] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5569.917146] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5579.233961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5579.242172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5579.257871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5582.310699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.311630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5583.662687] Lustre: server umount lustre-OST0000 complete [ 5588.961244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5588.972364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.999093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5589.001401] Lustre: Skipped 1 previous similar message [ 5592.544406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5592.544606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5594.088994] Lustre: server umount lustre-MDT0000 complete [ 5597.668313] LustreError: 170856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5597.695727] LustreError: 170856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5597.779422] LustreError: 170839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315995 with bad export cookie 8600107073095941993 [ 5597.793318] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5597.793782] LustreError: 170839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5598.318811] Lustre: server umount lustre-MDT0001 complete [ 5610.263863] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5612.817640] Key type lgssc unregistered [ 5613.079816] LNet: 174393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5613.087337] LNetError: 174393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5613.097989] LNet: Removed LNI 192.168.202.129@tcp [ 5613.890483] Key type .llcrypt unregistered [ 5613.891814] Key type ._llcrypt unregistered [ 5645.581558] Key type ._llcrypt registered [ 5645.584541] Key type .llcrypt registered [ 5645.791465] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5647.117394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5647.176953] alg: No test for adler32 (adler32-zlib) [ 5648.281732] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 5648.606585] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 5650.335203] Key type lgssc registered [ 5651.439632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5659.247365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5659.264321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.635534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.355524] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5665.762570] LustreError: 176095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5665.783775] LustreError: 176095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5670.889469] LustreError: 176096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5672.992928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.284933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5677.616691] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5684.458247] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5690.532333] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5697.043507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5697.453634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5705.047551] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5705.669075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5712.810271] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5724.144041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5724.151465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.177696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5726.181444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5726.189788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5726.191817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5727.516812] Lustre: server umount lustre-OST0000 complete [ 5734.880595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5734.895050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.936850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5734.951121] Lustre: Skipped 1 previous similar message [ 5738.088834] Lustre: server umount lustre-MDT0000 complete [ 5741.539871] LustreError: 176092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.551668] LustreError: 176092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5742.306322] LustreError: 176075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316139 with bad export cookie 10440841958080299062 [ 5742.312836] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5743.100497] Lustre: server umount lustre-MDT0001 complete [ 5755.762219] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5758.880760] Key type lgssc unregistered [ 5759.283216] LNet: 179632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5759.294440] LNetError: 179632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5759.320034] LNet: Removed LNI 192.168.202.129@tcp [ 5760.453179] Key type .llcrypt unregistered [ 5760.454777] Key type ._llcrypt unregistered [ 5787.480468] Key type ._llcrypt registered [ 5787.491531] Key type .llcrypt registered [ 5787.646920] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5788.671883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5788.854383] alg: No test for adler32 (adler32-zlib) [ 5790.053505] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 5790.272890] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 5792.007243] Key type lgssc registered [ 5793.367438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5803.572644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5803.583917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5804.885700] Lustre: *** cfs_fail_loc=135, val=0*** [ 5804.998730] LustreError: 181164:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5805.003780] LustreError: 181164:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.129@tcp: cfg command failed: rc = -2 [ 5805.010324] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5805.010324] [ 5805.016838] LustreError: MGC192.168.202.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5805.026986] LustreError: 181124:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5805.036445] LustreError: 181124:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5805.054319] LustreError: 181124:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5805.063890] LustreError: 181124:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5805.196406] Lustre: server umount lustre-MDT0000 complete [ 5805.205609] LustreError: 181124:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5811.597685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5812.124419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5817.181933] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5822.445606] LustreError: 181535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5822.461712] LustreError: 181535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5825.683426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.010366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5830.226888] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5837.553214] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5843.292038] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5846.502525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5846.513582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5846.530593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5847.526827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5847.527450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.566127] Lustre: Skipped 1 previous similar message [ 5851.617803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5851.622305] Lustre: Skipped 1 previous similar message [ 5856.738665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5856.749041] Lustre: Skipped 1 previous similar message [ 5859.296216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5859.487874] Lustre: server umount lustre-MDT0000 complete [ 5861.860944] LustreError: 182794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.879714] LustreError: 182794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5863.547418] LustreError: 181517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316260 with bad export cookie 9171213250102235692 [ 5863.551213] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5863.571370] LustreError: 181517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5864.042391] Lustre: server umount lustre-MDT0001 complete [ 5878.914936] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5883.182917] Key type lgssc unregistered [ 5883.455650] LNet: 183861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5883.459363] LNetError: 183861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5883.479593] LNet: Removed LNI 192.168.202.129@tcp [ 5884.067252] Key type .llcrypt unregistered [ 5884.069671] Key type ._llcrypt unregistered [ 5905.820727] Key type ._llcrypt registered [ 5905.823201] Key type .llcrypt registered [ 5905.929548] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5906.207586] Key type .llcrypt unregistered [ 5906.209224] Key type ._llcrypt unregistered [ 5922.531754] Key type ._llcrypt registered [ 5922.532823] Key type .llcrypt registered [ 5922.651763] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5923.873294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5924.101309] alg: No test for adler32 (adler32-zlib) [ 5925.141355] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 5925.458756] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 5927.183280] Key type lgssc registered [ 5928.686583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5937.824790] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5937.849960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5953.441569] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5958.562753] LustreError: 185997:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.129@tcp: failed processing log, type 4: rc = -110 [ 5985.248142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5991.973569] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5999.419205] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6005.571234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6006.103208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6006.174492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6011.357906] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6011.370784] LustreError: 186021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6011.403853] LustreError: 186021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6016.484825] LustreError: 186022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6020.577676] LustreError: 187035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6020.598734] LustreError: 187035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6020.878169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6021.357621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6026.094475] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6033.817808] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6039.631445] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6042.264899] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6052.342676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6052.360849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6052.366457] Lustre: Skipped 1 previous similar message [ 6052.369951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6052.395437] Lustre: Skipped 1 previous similar message [ 6057.461229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6057.473284] Lustre: Skipped 1 previous similar message [ 6062.561899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6062.573751] Lustre: Skipped 1 previous similar message [ 6065.119138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6065.490444] Lustre: server umount lustre-OST0000 complete [ 6072.799951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6072.804224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6072.822805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6072.839652] Lustre: Skipped 1 previous similar message [ 6072.861895] Lustre: Skipped 1 previous similar message [ 6076.086857] Lustre: server umount lustre-MDT0000 complete [ 6077.926937] LustreError: 187040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6077.948440] LustreError: 187040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6079.714876] LustreError: 188661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316477 with bad export cookie 13874253479711513356 [ 6079.716487] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6079.727764] LustreError: 188661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6080.454958] Lustre: server umount lustre-MDT0001 complete [ 6091.663781] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6095.511170] Key type lgssc unregistered [ 6095.912716] LNet: 189930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6095.930154] LNetError: 189930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6095.964221] LNet: Removed LNI 192.168.202.129@tcp [ 6097.008660] Key type .llcrypt unregistered [ 6097.011581] Key type ._llcrypt unregistered [ 6121.483578] Key type ._llcrypt registered [ 6121.486389] Key type .llcrypt registered [ 6121.618578] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6122.880546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6122.925124] alg: No test for adler32 (adler32-zlib) [ 6124.227727] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6124.523642] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6126.295894] Key type lgssc registered [ 6127.597325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6135.220450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6135.234914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6136.811106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6141.662820] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6147.040819] LustreError: 191342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6147.052986] LustreError: 191342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6152.163102] LustreError: 191341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.556620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6153.140697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6158.855677] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6167.147603] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6173.735576] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6182.158848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6182.571294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6189.040099] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6197.347360] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6207.157899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6211.951069] Lustre: Failing over lustre-MDT0000 [ 6212.249770] Lustre: server umount lustre-MDT0000 complete [ 6212.579035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6212.584649] LustreError: 191336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6212.600208] Lustre: Skipped 2 previous similar messages [ 6212.618995] LustreError: 191336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6217.697263] LustreError: 191336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6217.724899] LustreError: 191336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6222.826388] LustreError: 192070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6222.853299] LustreError: 192070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6227.935170] Lustre: 190653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786316609/real 1786316609] req@00000000fed53bec x1873088632352512/t0(0) o400->MGC192.168.202.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786316625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6227.964938] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6230.910278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6232.972100] LustreError: 191338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.009028] LustreError: 191338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6238.192780] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xba1ac5e9953fcc0 [ 6238.216975] Lustre: MGC192.168.202.129@tcp: Connection restored to 0@lo (at 0@lo) [ 6238.503060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6238.554262] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6240.074576] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6243.733461] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6243.835761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6243.897845] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6243.975044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6248.846716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6252.754551] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6261.735595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6261.749842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6261.769981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6263.071448] Lustre: server umount lustre-OST0000 complete [ 6269.409183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6269.417248] Lustre: Skipped 1 previous similar message [ 6269.422754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6269.423037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6269.449850] Lustre: Skipped 1 previous similar message [ 6274.313851] Lustre: server umount lustre-MDT0000 complete [ 6274.542594] LustreError: 192070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6274.571775] LustreError: 192070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6278.930026] LustreError: 191324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316676 with bad export cookie 838140527966944448 [ 6278.936575] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6278.939872] LustreError: 191324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6289.650743] Lustre: server umount lustre-MDT0001 complete [ 6302.752813] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6306.939587] Key type lgssc unregistered [ 6307.464253] LNet: 196165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6307.491941] LNetError: 196165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6307.526528] LNet: Removed LNI 192.168.202.129@tcp [ 6308.992396] Key type .llcrypt unregistered [ 6308.995274] Key type ._llcrypt unregistered [ 6336.448772] Key type ._llcrypt registered [ 6336.450599] Key type .llcrypt registered [ 6336.532520] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6337.685033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6337.752743] alg: No test for adler32 (adler32-zlib) [ 6338.875410] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6339.176131] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6340.905712] Key type lgssc registered [ 6342.225708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6351.326385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6351.341301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6352.821811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6357.711893] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6363.107139] LustreError: 197579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6363.116646] LustreError: 197579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6366.193396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.525513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6372.200153] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6379.875923] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6385.790801] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6392.421844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6392.948842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6398.080778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6398.393712] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6406.142523] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6409.003516] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6413.472813] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6428.428954] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6434.272575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6434.283919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6434.308364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6437.345381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6437.356236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6437.363202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.097523] Lustre: server umount lustre-OST0000 complete [ 6447.590103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6447.598859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6447.608252] Lustre: Skipped 2 previous similar messages [ 6451.022479] Lustre: server umount lustre-MDT0000 complete [ 6452.705234] LustreError: 198306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6452.726017] LustreError: 198306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6454.850727] LustreError: 200113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316852 with bad export cookie 5764280614799104268 [ 6454.856389] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6454.863977] LustreError: 200113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6455.423258] Lustre: server umount lustre-MDT0001 complete [ 6466.556308] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6469.221735] Key type lgssc unregistered [ 6469.708884] LNet: 201426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6469.717049] LNetError: 201426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6469.740708] LNet: Removed LNI 192.168.202.129@tcp [ 6470.835952] Key type .llcrypt unregistered [ 6470.837597] Key type ._llcrypt unregistered [ 6493.678893] Key type ._llcrypt registered [ 6493.680869] Key type .llcrypt registered [ 6493.830261] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6495.274444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6495.347091] alg: No test for adler32 (adler32-zlib) [ 6496.427839] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6496.623270] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6498.272586] Key type lgssc registered [ 6499.967590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6507.795420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6507.825362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6509.449069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6513.115800] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6518.762270] LustreError: 202694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6518.772253] LustreError: 202694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6521.441504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6521.874787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6527.072723] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6534.747972] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6539.920693] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6546.064947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6546.470604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6553.144548] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6553.650770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6558.946583] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6564.927752] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6577.839558] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6586.235487] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6595.260219] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6608.865277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6608.892218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6608.917212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6609.889709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6609.894862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6612.831371] Lustre: server umount lustre-OST0000 complete [ 6619.104539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6619.110462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6619.134528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6619.137028] Lustre: Skipped 1 previous similar message [ 6625.249322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6625.253299] Lustre: Skipped 3 previous similar messages [ 6630.367309] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6630.373203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6630.382054] Lustre: Skipped 1 previous similar message [ 6630.537687] Lustre: server umount lustre-MDT0000 complete [ 6633.988179] LustreError: 202674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317031 with bad export cookie 18175639122358057786 [ 6634.000802] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6634.004897] LustreError: 202674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6634.272885] Lustre: server umount lustre-MDT0001 complete [ 6644.963769] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6647.838248] Key type lgssc unregistered [ 6648.238611] LNet: 207769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6648.245786] LNetError: 207769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6648.261926] LNet: Removed LNI 192.168.202.129@tcp [ 6649.300327] Key type .llcrypt unregistered [ 6649.302933] Key type ._llcrypt unregistered [ 6678.649364] Key type ._llcrypt registered [ 6678.651575] Key type .llcrypt registered [ 6678.723081] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6679.586523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6679.604313] alg: No test for adler32 (adler32-zlib) [ 6680.640421] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6680.953668] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6682.663175] Key type lgssc registered [ 6684.059242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6692.812673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6692.828374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.232853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6698.519495] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6704.610984] LustreError: 209186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.627532] LustreError: 209186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6707.090631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6707.633543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6712.215559] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6719.373272] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6725.909506] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6733.919740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6734.334795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6740.328395] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6740.465233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6747.712925] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6760.942072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6760.945321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.960398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6761.953670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6761.961134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.978463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6767.046887] Lustre: server umount lustre-OST0000 complete [ 6772.194274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6772.200767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6772.226689] Lustre: Skipped 1 previous similar message [ 6772.247463] Lustre: Skipped 2 previous similar messages [ 6777.312378] LustreError: 209187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6777.329046] LustreError: 209187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6778.962455] Lustre: server umount lustre-MDT0000 complete [ 6782.440902] LustreError: 209913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6782.451172] LustreError: 209913:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6783.129961] LustreError: 209166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317180 with bad export cookie 151807533732277762 [ 6783.139745] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6783.150566] LustreError: 209166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6783.720534] Lustre: server umount lustre-MDT0001 complete [ 6795.698415] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6798.856875] Key type lgssc unregistered [ 6799.290928] LNet: 212873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6799.295867] LNetError: 212873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6799.325887] LNet: Removed LNI 192.168.202.129@tcp [ 6800.359093] Key type .llcrypt unregistered [ 6800.364148] Key type ._llcrypt unregistered [ 6820.845967] Key type ._llcrypt registered [ 6820.847767] Key type .llcrypt registered [ 6820.936223] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6822.292585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6822.310724] alg: No test for adler32 (adler32-zlib) [ 6823.434849] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6823.634567] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6825.311281] Key type lgssc registered [ 6826.598333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6834.464344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6834.489168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.930105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6840.354840] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6846.440874] LustreError: 214139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6846.458035] LustreError: 214139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6848.823236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6849.313764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6853.796262] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6861.581496] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6867.989267] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6875.086530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6875.510694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6881.656121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6882.792983] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6890.118070] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6904.289058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.297258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6904.329666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6904.344930] Lustre: Skipped 1 previous similar message [ 6909.410323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6909.427813] Lustre: Skipped 1 previous similar message [ 6910.713301] Lustre: server umount lustre-OST0000 complete [ 6916.064903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6916.087185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6916.100574] Lustre: Skipped 1 previous similar message [ 6916.104434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6919.648766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6919.650746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6919.666605] Lustre: Skipped 1 previous similar message [ 6921.426487] Lustre: server umount lustre-MDT0000 complete [ 6924.787177] LustreError: 214139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6924.818261] LustreError: 214139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6925.704279] LustreError: 214120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317323 with bad export cookie 5668034669832502779 [ 6925.711431] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6925.935788] Lustre: server umount lustre-MDT0001 complete [ 6938.693810] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6942.011532] Key type lgssc unregistered [ 6942.475665] LNet: 217827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6942.491418] LNetError: 217827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6942.521416] LNet: Removed LNI 192.168.202.129@tcp [ 6943.628322] Key type .llcrypt unregistered [ 6943.630044] Key type ._llcrypt unregistered [ 6964.410307] Key type ._llcrypt registered [ 6964.413870] Key type .llcrypt registered [ 6964.585391] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6966.537369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6966.597814] alg: No test for adler32 (adler32-zlib) [ 6967.578863] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6968.029338] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6969.855161] Key type lgssc registered [ 6971.674712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6979.870627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6979.890211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6981.307839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6985.016986] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6990.820517] LustreError: 219101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6990.849752] LustreError: 219101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6993.307289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6993.784767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6998.464441] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7005.817497] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7011.106216] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7017.926341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7018.251992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7024.720400] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7026.505580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7031.550471] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7041.211864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7041.278472] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7041.512123] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7048.617068] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7056.484739] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7058.048479] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7069.769654] Lustre: Permanently deactivating lustre-OST0001 [ 7069.782682] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7076.902317] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7082.986632] Lustre: Permanently reactivating lustre-OST0001 [ 7083.018393] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7083.020698] Lustre: Skipped 2 previous similar messages [ 7085.933317] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7085.956947] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7085.963807] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7085.985892] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7086.059522] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7098.857288] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.875180] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7098.881060] Lustre: Skipped 2 previous similar messages [ 7098.894752] Lustre: Skipped 1 previous similar message [ 7103.979677] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7103.999128] Lustre: Skipped 1 previous similar message [ 7109.108019] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7109.120136] Lustre: Skipped 1 previous similar message [ 7111.136527] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7111.181701] Lustre: server umount lustre-OST0001 complete [ 7114.239661] LustreError: 220921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7114.252920] LustreError: 220921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7116.769595] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7116.793626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7116.819827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7118.816282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7118.824058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7119.334693] LustreError: 220921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7120.752565] Lustre: server umount lustre-OST0000 complete [ 7127.011959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7127.024468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7127.039303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7127.041357] Lustre: Skipped 3 previous similar messages [ 7139.296746] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7139.735634] Lustre: server umount lustre-MDT0000 complete [ 7139.812786] LustreError: 220943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.834594] LustreError: 220943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7143.909757] LustreError: 219102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7143.946340] LustreError: 219102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7144.979160] LustreError: 219081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317542 with bad export cookie 13877739619747237005 [ 7144.983469] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7144.993684] LustreError: 219081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7145.564400] Lustre: server umount lustre-MDT0001 complete [ 7157.645853] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 7161.642161] Key type lgssc unregistered [ 7161.951699] LNet: 224172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7161.960457] LNetError: 224172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7162.989431] LNet: Removed LNI 192.168.202.129@tcp [ 7164.197788] Key type .llcrypt unregistered [ 7164.201547] Key type ._llcrypt unregistered [ 7202.450956] Key type ._llcrypt registered [ 7202.455225] Key type .llcrypt registered [ 7202.566878] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 7203.909619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7203.946884] alg: No test for adler32 (adler32-zlib) [ 7205.168883] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7205.503688] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 7207.215192] Key type lgssc registered [ 7208.239749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7216.237365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7216.263689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.581127] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7217.613886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7217.788416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7222.062114] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7228.873851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.934035] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7228.959991] Lustre: 226374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7229.158866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7233.446155] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7241.033611] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7247.349745] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7253.943256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7254.129311] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7254.287664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7259.824133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7260.451982] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7267.608354] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7271.536640] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7276.065614] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7282.905728] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7292.076766] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7309.576038] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7309.581664] Lustre: Skipped 1 previous similar message [ 7326.400286] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7326.405796] Lustre: Skipped 1 previous similar message [ 7360.042277] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7360.050234] Lustre: Skipped 3 previous similar messages [ 7430.792363] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7430.800500] Lustre: Skipped 8 previous similar messages [ 7454.180668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7454.191240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7454.211200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7454.689902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7454.705048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7454.730132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7457.173749] Lustre: server umount lustre-OST0000 complete [ 7464.930547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7464.932912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7464.940960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7464.946840] Lustre: Skipped 1 previous similar message [ 7464.962621] Lustre: Skipped 1 previous similar message [ 7467.082400] Lustre: server umount lustre-MDT0000 complete [ 7470.796147] LustreError: 229107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317868 with bad export cookie 3353128210535300020 [ 7470.801870] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7471.339767] Lustre: server umount lustre-MDT0001 complete [ 7483.515661] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 7486.532977] Key type lgssc unregistered [ 7486.762276] LNet: 231192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7486.769726] LNetError: 231192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7486.788473] LNet: Removed LNI 192.168.202.129@tcp [ 7488.029298] Key type .llcrypt unregistered [ 7488.031488] Key type ._llcrypt unregistered [ 7518.115925] Key type ._llcrypt registered [ 7518.120444] Key type .llcrypt registered [ 7518.242200] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 7519.949688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7520.061118] alg: No test for adler32 (adler32-zlib) [ 7521.429668] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7521.918437] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 7523.673098] Key type lgssc registered [ 7525.342695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7533.555206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7533.564209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.509441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7541.379483] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7544.817995] LustreError: 232610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7544.850896] LustreError: 232610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7549.923984] LustreError: 232609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7551.542329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7552.100242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7557.431789] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7568.019330] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7576.086408] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7583.481489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7584.063992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.980536] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7599.335169] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7605.570966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7623.138979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7623.147318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7623.168116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7623.594782] Lustre: server umount lustre-OST0000 complete [ 7628.769932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7628.778216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7628.793860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7631.339916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.341741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7631.357359] Lustre: Skipped 1 previous similar message [ 7636.448464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7636.460788] Lustre: Skipped 1 previous similar message [ 7637.547316] Lustre: server umount lustre-MDT0000 complete [ 7641.573164] LustreError: 235054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7641.954586] LustreError: 233343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318039 with bad export cookie 811074578699619734 [ 7641.982294] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7642.435661] Lustre: server umount lustre-MDT0001 complete [ 7654.835078] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 7658.649527] Key type lgssc unregistered [ 7658.850311] LNet: 236304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7658.859926] LNetError: 236304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7658.879145] LNet: Removed LNI 192.168.202.129@tcp [ 7659.931814] Key type .llcrypt unregistered [ 7659.934222] Key type ._llcrypt unregistered [ 7685.826692] Key type ._llcrypt registered [ 7685.829673] Key type .llcrypt registered [ 7685.987604] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 7687.888398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7687.935988] alg: No test for adler32 (adler32-zlib) [ 7689.179833] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7689.418849] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 7691.143931] Key type lgssc registered [ 7692.392345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7963.631279] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 7967.596208] Key type lgssc unregistered [ 7967.912505] LNet: 238336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7967.920637] LNetError: 238336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7967.959735] LNet: Removed LNI 192.168.202.129@tcp [ 7969.092768] Key type .llcrypt unregistered [ 7969.095471] Key type ._llcrypt unregistered [ 7996.625507] Key type ._llcrypt registered [ 7996.627409] Key type .llcrypt registered [ 7996.741340] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 8014.592606] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 8015.693463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8015.796731] alg: No test for adler32 (adler32-zlib) [ 8016.781601] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 8017.232033] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 8018.911722] Key type lgssc registered [ 8020.572640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8056.224369] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7812 sec ========== 19:34:11 (1786318451) [ 8058.262972] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:34:14 (1786318454) === [ 8062.186475] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:34:18 (1786318458) === [ 8106.798442] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 8110.055421] Key type lgssc unregistered [ 8110.361146] LNet: 243579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8110.368462] LNetError: 243579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8110.401208] LNet: Removed LNI 192.168.202.129@tcp [ 8111.506748] Key type .llcrypt unregistered [ 8111.514098] Key type ._llcrypt unregistered