[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 467599186 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002419] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006839] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009150] LSM: Security Framework initializing [ 0.010075] Yama: becoming mindful. [ 0.011040] SELinux: Initializing. [ 0.012084] *** VALIDATE selinux *** [ 0.020524] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025251] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027088] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028121] *** VALIDATE tmpfs *** [ 0.030169] *** VALIDATE proc *** [ 0.031174] *** VALIDATE cgroup *** [ 0.032006] *** VALIDATE cgroup2 *** [ 0.033003] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034121] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036020] Spectre V2 : User space: Vulnerable [ 0.037005] Speculative Store Bypass: Vulnerable [ 0.039960] debug: unmapping init [mem 0xffffffff8e259000-0xffffffff8e260fff] [ 0.041149] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042640] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043029] ... version: 2 [ 0.044017] ... bit width: 48 [ 0.045012] ... generic registers: 4 [ 0.046015] ... value mask: 0000ffffffffffff [ 0.047008] ... max period: 00007fffffffffff [ 0.048007] ... fixed-purpose events: 3 [ 0.048900] ... event mask: 000000070000000f [ 0.049293] rcu: Hierarchical SRCU implementation. [ 0.051421] smp: Bringing up secondary CPUs ... [ 0.052438] x86: Booting SMP configuration: [ 0.053023] .... node #0, CPUs: #1 #2 #3 [ 0.056075] smp: Brought up 1 node, 4 CPUs [ 0.057972] smpboot: Max logical packages: 1 [ 0.058011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.084054] node 0 deferred pages initialised in 23ms [ 0.087117] devtmpfs: initialized [ 0.088211] x86/mm: Memory block size: 128MB [ 0.090546] gcov: version magic: 0x41383552 [ 0.093096] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.095070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.097288] pinctrl core: initialized pinctrl subsystem [ 0.098125] [ 0.098559] ************************************************************* [ 0.100009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.102009] ** ** [ 0.105014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.107039] ** ** [ 0.109011] ** This means that this kernel is built to expose internal ** [ 0.111008] ** IOMMU data structures, which may compromise security on ** [ 0.113010] ** your system. ** [ 0.115010] ** ** [ 0.117009] ** If you see this message and you are not debugging the ** [ 0.119011] ** kernel, report this immediately to your vendor! ** [ 0.121013] ** ** [ 0.123010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.124007] ************************************************************* [ 0.126598] NET: Registered protocol family 16 [ 0.128307] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.130072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.133057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.136689] cpuidle: using governor menu [ 0.138725] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.140465] PCI: Using configuration type 1 for base access [ 0.142102] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.149119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.150012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.151260] cryptd: max_cpu_qlen set to 1000 [ 0.154217] ACPI: Added _OSI(Module Device) [ 0.155037] ACPI: Added _OSI(Processor Device) [ 0.156007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.156968] ACPI: Added _OSI(Processor Aggregator Device) [ 0.159000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.164470] ACPI: Interpreter enabled [ 0.165053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.167014] ACPI: Using IOAPIC for interrupt routing [ 0.168094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.170371] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.177000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.180030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.181013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.184088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.187027] acpiphp: Slot [2] registered [ 0.188032] acpiphp: Slot [5] registered [ 0.188987] acpiphp: Slot [6] registered [ 0.190096] acpiphp: Slot [7] registered [ 0.191085] acpiphp: Slot [8] registered [ 0.192123] acpiphp: Slot [9] registered [ 0.194087] acpiphp: Slot [10] registered [ 0.195091] acpiphp: Slot [3] registered [ 0.196063] acpiphp: Slot [4] registered [ 0.197014] acpiphp: Slot [11] registered [ 0.198164] acpiphp: Slot [12] registered [ 0.199096] acpiphp: Slot [13] registered [ 0.200061] acpiphp: Slot [14] registered [ 0.201059] acpiphp: Slot [15] registered [ 0.201993] acpiphp: Slot [16] registered [ 0.203056] acpiphp: Slot [17] registered [ 0.204061] acpiphp: Slot [18] registered [ 0.205006] acpiphp: Slot [19] registered [ 0.205933] acpiphp: Slot [20] registered [ 0.207085] acpiphp: Slot [21] registered [ 0.208104] acpiphp: Slot [22] registered [ 0.208982] acpiphp: Slot [23] registered [ 0.210072] acpiphp: Slot [24] registered [ 0.210850] acpiphp: Slot [25] registered [ 0.211093] acpiphp: Slot [26] registered [ 0.211980] acpiphp: Slot [27] registered [ 0.213074] acpiphp: Slot [28] registered [ 0.213999] acpiphp: Slot [29] registered [ 0.215094] acpiphp: Slot [30] registered [ 0.217121] acpiphp: Slot [31] registered [ 0.219079] PCI host bridge to bus 0000:00 [ 0.220019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.221020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.223015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.225018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.227017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.228014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.229123] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.231715] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.233823] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.239956] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.244052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.245014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.247026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.250017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.251576] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.254468] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.255050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.257528] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.261014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.271014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.274906] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.280275] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.288017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.296016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.310024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.317190] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.322019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.329016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.345023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.353982] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.357016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.366015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.378017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.388220] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.395019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.403020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.425018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.435056] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.442018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.450018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.469019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.480308] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.487019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.497017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.515020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.525492] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.528441] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.531411] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.533362] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.535235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.540021] iommu: Default domain type: Passthrough [ 0.542461] SCSI subsystem initialized [ 0.544172] ACPI: bus type USB registered [ 0.545092] usbcore: registered new interface driver usbfs [ 0.547091] usbcore: registered new interface driver hub [ 0.548087] usbcore: registered new device driver usb [ 0.550166] pps_core: LinuxPPS API ver. 1 registered [ 0.551009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.554067] PTP clock support registered [ 0.555124] EDAC MC: Ver: 3.0.0 [ 0.557114] PCI: Using ACPI for IRQ routing [ 0.558886] NetLabel: Initializing [ 0.560010] NetLabel: domain hash size = 128 [ 0.562009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.564075] NetLabel: unlabeled traffic allowed by default [ 0.566134] vgaarb: loaded [ 0.567258] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.569015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.575000] clocksource: Switched to clocksource kvm-clock [ 0.688213] VFS: Disk quotas dquot_6.6.0 [ 0.689684] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.692550] *** VALIDATE ramfs *** [ 0.693868] *** VALIDATE hugetlbfs *** [ 0.695283] pnp: PnP ACPI init [ 0.697662] pnp: PnP ACPI: found 6 devices [ 0.722539] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.726067] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.727950] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.729713] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.731887] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.733773] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.736085] NET: Registered protocol family 2 [ 0.738106] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.742307] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.745471] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.749568] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.753066] TCP: Hash tables configured (established 65536 bind 65536) [ 0.755791] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.758497] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.760738] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.763379] NET: Registered protocol family 1 [ 0.765232] RPC: Registered named UNIX socket transport module. [ 0.767109] RPC: Registered udp transport module. [ 0.769070] RPC: Registered tcp transport module. [ 0.770538] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.772579] NET: Registered protocol family 44 [ 0.774169] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.776214] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.778068] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.780746] PCI: CLS 0 bytes, default 64 [ 0.782306] Unpacking initramfs... [ 2.145184] debug: unmapping init [mem 0xffff93fdfcc54000-0xffff93fdfffbffff] [ 2.149340] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.151104] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.153112] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.633040] Initialise system trusted keyrings [ 2.634657] Key type blacklist registered [ 2.636545] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.643907] zbud: loaded [ 2.646324] *** VALIDATE nfs *** [ 2.647202] *** VALIDATE nfs4 *** [ 2.648559] pstore: using deflate compression [ 2.651782] Platform Keyring initialized [ 2.744631] NET: Registered protocol family 38 [ 2.745977] Key type asymmetric registered [ 2.746883] Asymmetric key parser 'x509' registered [ 2.748123] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.750389] io scheduler mq-deadline registered [ 2.751521] io scheduler kyber registered [ 2.752521] io scheduler bfq registered [ 2.753693] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.755576] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.757465] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.759365] ACPI: Power Button [PWRF] [ 2.762737] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.767521] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.777257] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.783338] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.793026] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.818202] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.843122] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.846338] Non-volatile memory driver v1.3 [ 2.847399] Linux agpgart interface v0.103 [ 2.871024] virtio_blk virtio1: [vda] 149784 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.872787] vda: detected capacity change from 0 to 76689408 [ 2.887620] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.889367] vdb: detected capacity change from 0 to 1073741824 [ 2.903317] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.905067] vdc: detected capacity change from 0 to 2621440000 [ 2.928182] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.930265] vdd: detected capacity change from 0 to 2621440000 [ 2.953861] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.955775] vde: detected capacity change from 0 to 4294967296 [ 2.969456] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.971283] vdf: detected capacity change from 0 to 4294967296 [ 2.975980] libphy: Fixed MDIO Bus: probed [ 2.983743] usbcore: registered new interface driver usbserial_generic [ 2.986863] usbserial: USB Serial support registered for generic [ 2.990366] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.995573] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.997525] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.000277] mousedev: PS/2 mouse device common for all mice [ 3.003324] rtc_cmos 00:05: RTC can wake from S4 [ 3.003523] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.005821] rtc_cmos 00:05: registered as rtc0 [ 3.010321] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.014444] intel_pstate: CPU model not supported [ 3.017437] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.020279] hid: raw HID events driver (C) Jiri Kosina [ 3.026792] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.027874] usbcore: registered new interface driver usbhid [ 3.031647] usbhid: USB HID core driver [ 3.032924] drop_monitor: Initializing network drop monitor service [ 3.034956] Initializing XFRM netlink socket [ 3.036965] NET: Registered protocol family 10 [ 3.039781] Segment Routing with IPv6 [ 3.041215] NET: Registered protocol family 17 [ 3.043357] mpls_gso: MPLS GSO support [ 3.047664] RAS: Correctable Errors collector initialized. [ 3.049491] AVX version of gcm_enc/dec engaged. [ 3.051286] AES CTR mode by8 optimization enabled [ 3.134433] sched_clock: Marking stable (3134401316, 0)->(3968751447, -834350131) [ 3.138350] registered taskstats version 1 [ 3.140593] Loading compiled-in X.509 certificates [ 3.142127] zswap: loaded using pool lzo/zbud [ 3.162951] Key type big_key registered [ 3.174613] Key type encrypted registered [ 3.176653] ima: No TPM chip found, activating TPM-bypass! [ 3.178873] ima: Allocated hash algorithm: sha1 [ 3.180697] ima: No architecture policies found [ 3.182747] evm: Initialising EVM extended attributes: [ 3.184836] evm: security.selinux [ 3.186313] evm: security.ima [ 3.187575] evm: security.capability [ 3.189064] evm: HMAC attrs: 0x1 [ 3.191480] rtc_cmos 00:05: setting system clock to 2026-09-07 16:39:52 UTC (1788799192) [ 3.198281] debug: unmapping init [mem 0xffffffff8f203000-0xffffffff8f3fffff] [ 3.201690] debug: unmapping init [mem 0xffffffff8df82000-0xffffffff8e258fff] [ 3.211542] Write protecting the kernel read-only data: 28672k [ 3.214896] debug: unmapping init [mem 0xffffffff8c603000-0xffffffff8c7fffff] [ 3.217789] debug: unmapping init [mem 0xffffffff8cf14000-0xffffffff8cffffff] [ 3.250710] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.258574] systemd[1]: Detected virtualization kvm. [ 3.260639] systemd[1]: Detected architecture x86-64. [ 3.262609] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.289767] systemd[1]: No hostname configured. [ 3.291666] systemd[1]: Set hostname to . [ 3.293889] random: systemd: uninitialized urandom read (16 bytes read) [ 3.296217] systemd[1]: Initializing machine ID from random generator. [ 3.320727] random: ln: uninitialized urandom read (6 bytes read) [ 3.414093] random: systemd: uninitialized urandom read (16 bytes read) [ 3.416802] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.421250] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.429119] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.971501] device-mapper: uevent: version 1.0.3 [ 3.973922] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 4.601241] random: fast init done Started Hardware RNG Entropy Gatherer Daemon. [ 4.640316] virtio_net virtio0 ens2: renamed from eth0 [ 4.650907] scsi host0: ata_piix [ 4.696226] scsi host1: ata_piix [ 4.698504] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.700788] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.262681] dracut-initqueue[576]: RTNETLINK answers: File exists [ 9.694803] random: crng init done [ 9.696042] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.026753] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.058584] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.266853] SELinux: Disabled at runtime. [ 11.327086] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.335747] systemd[1]: Detected virtualization kvm. [ 11.337706] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.774818] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.778329] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.783713] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.787744] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.789929] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.796930] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.800915] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 11.855510] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.219571] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.458849] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.482633] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.645625] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.660739] EDAC sbridge: Ver: 1.1.2 [ 14.395859] Key type dns_resolver registered [ 14.701298] NFS: Registering the id_resolver key type [ 14.703037] Key type id_resolver registered [ 14.704333] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg208-server login: [ 36.491965] hrtimer: interrupt took 2102178 ns [ 84.199667] libcfs: loading out-of-tree module taints kernel. [ 84.243783] Key type ._llcrypt registered [ 84.245792] Key type .llcrypt registered [ 84.370914] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 105.035565] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 107.377476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.389494] alg: No test for adler32 (adler32-zlib) [ 109.368320] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 110.357857] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 112.168397] Key type lgssc registered [ 114.455947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.140460] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 184.652163] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 201.062852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 201.104338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.716851] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 202.783056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 202.863263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 203.080913] Lustre: lustre-MDT0000: new disk, initializing [ 203.196884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.257201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 209.839479] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.526444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.666316] Lustre: 6506:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 224.707951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 224.718256] Lustre: Skipped 1 previous similar message [ 224.826567] Lustre: lustre-MDT0001: new disk, initializing [ 224.906229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 224.964104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 224.995049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 231.694866] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.397568] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 250.639281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.012469] Lustre: lustre-OST0000: new disk, initializing [ 251.017807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 251.026101] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.104212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 253.301104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 253.339950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 253.495594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 256.546564] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.436281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 273.653754] Lustre: lustre-OST0001: new disk, initializing [ 273.658723] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 273.665351] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 273.769901] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 279.590023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 279.603911] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 279.753249] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 282.606725] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 295.052554] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 303.780368] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 311.293662] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing check_logdir /tmp/testlogs/ [ 317.631771] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing yml_node [ 323.400808] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 326.647128] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 329.802928] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 331.407313] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 12:45:19 EDT 2026 [ 352.585673] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 354.679662] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 373.728722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 373.746089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.783267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 377.312920] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.315412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 377.331809] Lustre: Skipped 2 previous similar messages [ 377.348904] Lustre: Skipped 3 previous similar messages [ 379.240651] Lustre: server umount lustre-MDT0000 complete [ 383.523826] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799572 with bad export cookie 3696194770815867680 [ 383.528653] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 383.534660] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 387.552982] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 387.553633] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 387.556784] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 387.573783] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 387.584407] Lustre: Skipped 1 previous similar message [ 388.576018] LustreError: 10096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.598582] LustreError: 10096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 390.048779] Lustre: server umount lustre-MDT0001 complete [ 394.973405] Lustre: server umount lustre-OST0000 complete [ 399.882952] Lustre: server umount lustre-OST0001 complete [ 407.866709] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 419.030671] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 468.338506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.646979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 468.692166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 468.813718] Lustre: lustre-MDT0000: new disk, initializing [ 468.923957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.958151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 474.019844] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 486.511925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.671877] Lustre: 15076:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 486.731274] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 486.735294] Lustre: Skipped 1 previous similar message [ 486.863673] Lustre: lustre-MDT0001: new disk, initializing [ 486.989124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 487.040033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 487.055959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 492.888649] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 503.855767] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 510.903957] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 518.571802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 518.929423] Lustre: lustre-OST0000: new disk, initializing [ 518.941401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 518.967211] Lustre: 17046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 519.121218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.825805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 520.837915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 520.995567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 526.139702] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 538.179612] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 544.386789] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 544.686382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 550.105150] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 550.312751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 557.026179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 557.036285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.047379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 557.060286] Lustre: Skipped 1 previous similar message [ 557.076134] Lustre: Skipped 4 previous similar messages [ 562.180068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 562.186959] Lustre: Skipped 1 previous similar message [ 567.267210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 567.467072] Lustre: server umount lustre-OST0000 complete [ 572.388970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.396420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 572.418468] Lustre: Skipped 1 previous similar message [ 572.449725] Lustre: Skipped 3 previous similar messages [ 578.549691] Lustre: server umount lustre-MDT0000 complete [ 582.626622] LustreError: 16302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.635533] LustreError: 16302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 582.864584] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799772 with bad export cookie 3696194770815869213 [ 582.871367] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 582.876363] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 583.184703] Lustre: server umount lustre-MDT0001 complete [ 586.520296] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 12:49:34 (1788799774) [ 594.084886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.563815] LustreError: 19311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.637437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 600.035314] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.043618] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 609.277335] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.329638] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 610.772693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.414564] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 625.520221] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 632.531139] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 640.517828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 648.870598] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 657.330147] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 679.911302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 679.926217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 679.928102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 679.928110] Lustre: Skipped 2 previous similar messages [ 679.948143] Lustre: Skipped 1 previous similar message [ 683.592287] Lustre: server umount lustre-OST0000 complete [ 695.915362] Lustre: server umount lustre-MDT0000 complete [ 700.393634] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.422536] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 701.473074] LustreError: 19292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799890 with bad export cookie 3696194770815870095 [ 701.501947] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.502316] LustreError: 19292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 702.055963] Lustre: server umount lustre-MDT0001 complete [ 717.348809] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 721.059124] Key type lgssc unregistered [ 721.491720] LNet: 22868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.498793] LNetError: 22868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.560264] LNet: Removed LNI 192.168.202.108@tcp [ 723.797475] Key type .llcrypt unregistered [ 723.800389] Key type ._llcrypt unregistered [ 751.917936] Key type ._llcrypt registered [ 751.920943] Key type .llcrypt registered [ 752.140962] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 754.048285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 754.065068] alg: No test for adler32 (adler32-zlib) [ 755.195624] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 755.545650] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 757.343150] Key type lgssc registered [ 759.281903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 770.068119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 770.084949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 771.739031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 778.439996] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 782.318256] LustreError: 24280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 782.362637] LustreError: 24280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 787.434316] LustreError: 24281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 788.986908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.447155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 795.780216] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 805.080322] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 811.744894] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 819.721707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 820.329244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.025848] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 837.001548] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 841.853513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 856.547665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 856.554654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.571540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 857.572681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 857.574168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 857.592244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.789800] Lustre: server umount lustre-OST0000 complete [ 867.810180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 867.812612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 867.831574] Lustre: Skipped 1 previous similar message [ 867.844939] Lustre: Skipped 2 previous similar messages [ 872.856164] Lustre: server umount lustre-MDT0000 complete [ 872.940126] LustreError: 25541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 872.963849] LustreError: 25541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 876.864725] LustreError: 26126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800066 with bad export cookie 10741846501790268399 [ 876.875269] LustreError: 26126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 876.875657] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 877.320469] Lustre: server umount lustre-MDT0001 complete [ 888.952980] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 891.889864] Key type lgssc unregistered [ 892.209237] LNet: 28255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.216848] LNetError: 28255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.231677] LNet: Removed LNI 192.168.202.108@tcp [ 893.099725] Key type .llcrypt unregistered [ 893.102099] Key type ._llcrypt unregistered [ 918.485074] Key type ._llcrypt registered [ 918.487944] Key type .llcrypt registered [ 918.625829] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 919.883769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 919.979880] alg: No test for adler32 (adler32-zlib) [ 921.184252] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 921.525848] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 923.200265] Key type lgssc registered [ 924.699501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 933.493241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 933.507889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.874157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 939.624843] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 945.128290] LustreError: 29667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 945.141812] LustreError: 29667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 948.322589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 948.796899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 954.215630] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 962.534494] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 969.684266] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 985.069542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 985.434053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.264455] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1001.839469] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1004.966180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1019.365627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1019.368441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.378459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.396865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1020.398867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.425256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.448355] Lustre: Skipped 1 previous similar message [ 1023.698481] Lustre: server umount lustre-OST0000 complete [ 1030.628117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1030.637140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1030.641237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1030.656686] Lustre: Skipped 1 previous similar message [ 1034.304637] Lustre: server umount lustre-MDT0000 complete [ 1035.750798] LustreError: 29668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.788346] LustreError: 29668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1039.377742] LustreError: 29649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800228 with bad export cookie 4606052350604273311 [ 1039.380888] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1039.393519] LustreError: 29649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1039.778960] Lustre: server umount lustre-MDT0001 complete [ 1053.471871] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1056.439333] Key type lgssc unregistered [ 1056.767860] LNet: 33568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.782452] LNetError: 33568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.840596] LNet: Removed LNI 192.168.202.108@tcp [ 1058.957711] Key type .llcrypt unregistered [ 1058.959758] Key type ._llcrypt unregistered [ 1086.691437] Key type ._llcrypt registered [ 1086.694168] Key type .llcrypt registered [ 1086.828584] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1088.458596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1088.657607] alg: No test for adler32 (adler32-zlib) [ 1089.799418] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 1090.188080] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1091.921121] Key type lgssc registered [ 1093.744264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1103.161789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1103.185727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1104.738226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.771294] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1115.108247] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.121311] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1119.525940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.058529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1125.670595] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1134.508410] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1142.701866] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1149.754759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1149.930385] LustreError: 36798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.088579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.585922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1157.843591] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1168.937526] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1181.160402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1181.169523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.196355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1185.248143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1185.252311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1185.265166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1185.315677] Lustre: Skipped 1 previous similar message [ 1186.689525] Lustre: server umount lustre-OST0000 complete [ 1191.909334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1191.924692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.945480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1195.496230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1195.514543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.610793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.623226] Lustre: Skipped 2 previous similar messages [ 1205.215134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1205.382261] Lustre: server umount lustre-MDT0000 complete [ 1209.353457] LustreError: 34959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800398 with bad export cookie 4946904754548688896 [ 1209.362589] LustreError: 34959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1209.363117] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1209.723926] Lustre: server umount lustre-MDT0001 complete [ 1225.495664] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1229.189437] Key type lgssc unregistered [ 1229.635142] LNet: 38513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.647662] LNetError: 38513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.669735] LNet: Removed LNI 192.168.202.108@tcp [ 1230.489193] Key type .llcrypt unregistered [ 1230.499066] Key type ._llcrypt unregistered [ 1257.652444] Key type ._llcrypt registered [ 1257.654183] Key type .llcrypt registered [ 1257.871645] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1260.090860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1260.257074] alg: No test for adler32 (adler32-zlib) [ 1261.364424] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 1261.596331] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1263.288456] Key type lgssc registered [ 1265.535734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1276.954759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1276.988794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1278.755708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.606371] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1289.214538] LustreError: 39947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.252110] LustreError: 39947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1294.304639] LustreError: 39948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.764151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.273374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1300.270268] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1309.430425] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1316.835231] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1324.613474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1325.058847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.313913] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1342.245682] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1349.634850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1360.867522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1360.892189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.913402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1364.786530] Lustre: lustre-OST0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 1364.968895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1364.979924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1366.834968] Lustre: server umount lustre-OST0000 complete [ 1390.371326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.392219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.394870] Lustre: Skipped 1 previous similar message [ 1390.431397] Lustre: Skipped 3 previous similar messages [ 1394.735785] Lustre: server umount lustre-MDT0000 complete [ 1395.683959] LustreError: 42439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.722994] LustreError: 42439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1398.652820] LustreError: 39929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800587 with bad export cookie 18418169426481027465 [ 1398.668790] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1398.670419] LustreError: 39929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1405.131058] Lustre: server umount lustre-MDT0001 complete [ 1416.887315] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1419.946268] Key type lgssc unregistered [ 1420.288211] LNet: 43681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.303340] LNetError: 43681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.320759] LNet: Removed LNI 192.168.202.108@tcp [ 1421.221215] Key type .llcrypt unregistered [ 1421.229462] Key type ._llcrypt unregistered [ 1446.887777] Key type ._llcrypt registered [ 1446.890279] Key type .llcrypt registered [ 1447.025304] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1448.119935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1448.494590] alg: No test for adler32 (adler32-zlib) [ 1449.540718] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 1449.773116] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1451.439169] Key type lgssc registered [ 1452.898920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1461.274628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1461.320683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.839641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1466.905796] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1472.998719] LustreError: 45093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.029113] LustreError: 45093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1475.463440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.793733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1480.258650] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1488.691494] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1495.087035] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1502.715746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1503.225673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1508.357124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1510.482340] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1517.101818] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1526.755605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.764871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1526.767102] Lustre: Skipped 1 previous similar message [ 1526.774217] Lustre: Skipped 2 previous similar messages [ 1529.123810] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 1531.877103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1531.890996] Lustre: Skipped 2 previous similar messages [ 1532.681850] Lustre: server umount lustre-MDT0000 complete [ 1534.196401] LustreError: 47583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1536.539662] LustreError: 45073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800725 with bad export cookie 7318656848660788184 [ 1536.544191] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1536.548951] LustreError: 45073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1536.992406] LustreError: 45822:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bc07193b x1875693418870016/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1536.999215] LustreError: 47674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1537.004337] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1537.004347] Lustre: Skipped 1 previous similar message [ 1537.007578] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1539.322076] LustreError: 45087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.112110] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1542.121116] Lustre: Skipped 1 previous similar message [ 1543.030966] Lustre: server umount lustre-MDT0001 complete [ 1558.616362] Lustre: server umount lustre-OST0000 complete [ 1576.185615] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1578.761329] Key type lgssc unregistered [ 1579.045227] LNet: 48952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.049645] LNetError: 48952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.063683] LNet: Removed LNI 192.168.202.108@tcp [ 1579.700097] Key type .llcrypt unregistered [ 1579.701861] Key type ._llcrypt unregistered [ 1607.216894] Key type ._llcrypt registered [ 1607.219445] Key type .llcrypt registered [ 1607.300288] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1608.540601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1608.701398] alg: No test for adler32 (adler32-zlib) [ 1609.792385] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 1610.091465] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1611.783235] Key type lgssc registered [ 1613.229433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1621.291539] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1621.316470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1636.960718] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1642.079360] LustreError: 50351:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 1668.767302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.069897] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1685.381980] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1850.190858] Lustre: server umount lustre-OST0000 complete [ 1869.602948] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1873.549346] Key type lgssc unregistered [ 1873.959530] LNet: 52026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.968196] LNetError: 52026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.982070] LNet: Removed LNI 192.168.202.108@tcp [ 1876.322701] Key type .llcrypt unregistered [ 1876.342538] Key type ._llcrypt unregistered [ 1905.215145] Key type ._llcrypt registered [ 1905.217826] Key type .llcrypt registered [ 1905.355857] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1907.014680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1907.226759] alg: No test for adler32 (adler32-zlib) [ 1908.413464] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 1908.901132] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1910.679241] Key type lgssc registered [ 1912.128429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1922.323720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1922.340198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.752834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.152438] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1934.308124] LustreError: 53442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.334325] LustreError: 53442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1938.128716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.624957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1943.085340] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1954.105870] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1961.979190] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1970.767830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1971.091691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.342947] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1990.417970] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1993.699071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1993.703269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.749745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1994.795437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1999.849114] LustreError: 55265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.089668] Lustre: server umount lustre-OST0000 complete [ 2010.086354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2010.095905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.106245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.106255] Lustre: Skipped 1 previous similar message [ 2010.181540] Lustre: Skipped 1 previous similar message [ 2011.873782] Lustre: server umount lustre-MDT0000 complete [ 2015.211267] LustreError: 53443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.235037] LustreError: 53443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2016.752497] LustreError: 53423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801206 with bad export cookie 3497842491386631126 [ 2016.758779] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2016.765253] LustreError: 53423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2017.335253] Lustre: server umount lustre-MDT0001 complete [ 2033.052306] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2036.651386] Key type lgssc unregistered [ 2037.009661] LNet: 56973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.026764] LNetError: 56973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.053746] LNet: Removed LNI 192.168.202.108@tcp [ 2039.279886] Key type .llcrypt unregistered [ 2039.282296] Key type ._llcrypt unregistered [ 2067.695815] Key type ._llcrypt registered [ 2067.701218] Key type .llcrypt registered [ 2067.835979] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2069.004237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2069.080767] alg: No test for adler32 (adler32-zlib) [ 2070.197320] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 2070.523181] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2072.312833] Key type lgssc registered [ 2074.142284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2084.080371] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2084.126928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2099.812268] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2104.927401] LustreError: 58357:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 2131.615778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2140.922687] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2151.388375] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2162.438639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.298729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.417977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2169.794206] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2171.876961] LustreError: 58382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.896748] LustreError: 58382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2177.002995] LustreError: 58383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.039122] LustreError: 58383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2182.118215] LustreError: 59398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.158482] LustreError: 59398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2182.577466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.341818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2190.493920] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2201.102837] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2208.864425] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2213.867917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2213.878489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.888076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.377113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2218.162318] Lustre: server umount lustre-OST0000 complete [ 2244.074498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.088263] Lustre: Skipped 1 previous similar message [ 2244.096929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2244.104179] Lustre: Skipped 3 previous similar messages [ 2249.193764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.221772] Lustre: Skipped 1 previous similar message [ 2254.308853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2254.316439] Lustre: Skipped 1 previous similar message [ 2257.375410] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2257.639230] Lustre: server umount lustre-MDT0000 complete [ 2259.430506] LustreError: 59902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.460984] LustreError: 59902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2262.076282] LustreError: 58365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801451 with bad export cookie 470310762907522564 [ 2262.081395] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.093200] LustreError: 58365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2268.694775] Lustre: server umount lustre-MDT0001 complete [ 2282.973569] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2286.723398] Key type lgssc unregistered [ 2287.237229] LNet: 62111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.243510] LNetError: 62111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.294905] LNet: Removed LNI 192.168.202.108@tcp [ 2289.487126] Key type .llcrypt unregistered [ 2289.489763] Key type ._llcrypt unregistered [ 2318.888655] Key type ._llcrypt registered [ 2318.890858] Key type .llcrypt registered [ 2319.029141] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2320.436179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2320.631919] alg: No test for adler32 (adler32-zlib) [ 2321.925944] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 2322.087138] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2323.815261] Key type lgssc registered [ 2325.354823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2335.705898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2335.725628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.329355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.101543] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2347.499076] LustreError: 63523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.518343] LustreError: 63523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2352.617901] LustreError: 63522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.770772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.163881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2360.066785] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2369.328438] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2375.991703] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2384.231957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2384.710878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.816699] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2401.537505] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2407.241660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2419.680937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2419.687809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.717472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2422.758041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2422.758574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2422.793681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.786380] Lustre: server umount lustre-OST0000 complete [ 2432.994888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2433.002235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2433.015969] Lustre: Skipped 1 previous similar message [ 2433.051374] Lustre: Skipped 2 previous similar messages [ 2438.121629] LustreError: 63522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.172835] Lustre: server umount lustre-MDT0000 complete [ 2442.864357] LustreError: 63504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801632 with bad export cookie 7847702609650282617 [ 2442.874022] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2442.880130] LustreError: 63504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2443.490744] Lustre: server umount lustre-MDT0001 complete [ 2457.054840] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2460.791573] Key type lgssc unregistered [ 2461.149613] LNet: 67077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2461.175302] LNetError: 67077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2462.246911] LNet: Removed LNI 192.168.202.108@tcp [ 2463.395520] Key type .llcrypt unregistered [ 2463.399260] Key type ._llcrypt unregistered [ 2492.768243] Key type ._llcrypt registered [ 2492.769833] Key type .llcrypt registered [ 2492.864076] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2493.929819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2494.118304] alg: No test for adler32 (adler32-zlib) [ 2495.182107] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 2495.397553] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2497.106361] Key type lgssc registered [ 2498.567667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2508.254369] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 13:21:36 (1788801696) [ 2516.539328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2516.562387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.101806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.588784] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2528.233117] LustreError: 68777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.262755] LustreError: 68777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2533.348525] LustreError: 68778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.111089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2534.593039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2539.646193] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2548.061835] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2554.685422] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2562.695695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2563.344579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2564.343344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2571.395529] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2580.864639] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2587.824690] Lustre: Failing over lustre-MDT0000 [ 2588.128060] Lustre: server umount lustre-MDT0000 complete [ 2588.428122] LustreError: 68772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.183632] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.195274] Lustre: Skipped 1 previous similar message [ 2593.536489] LustreError: 68773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.574991] LustreError: 68773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2598.649418] LustreError: 68772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.664787] LustreError: 68772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2601.496719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.734872] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2602.018297] Lustre: *** cfs_fail_loc=135, val=0*** [ 2602.143567] LustreError: 71864:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2602.155534] LustreError: 71864:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.108@tcp: cfg command failed: rc = -2 [ 2602.168390] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2602.168390] [ 2602.174831] LustreError: MGC192.168.202.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2602.188511] LustreError: 71852:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2602.198296] LustreError: 71852:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2602.215596] LustreError: 71852:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2602.230933] LustreError: 71852:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2602.271376] Lustre: server umount lustre-MDT0000 complete [ 2602.275154] LustreError: 71852:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2609.918386] LustreError: 68772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.972658] LustreError: 68772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2611.145873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.267931] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2611.276729] LustreError: 68769:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2611.740761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.840438] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2615.031598] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2616.834976] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2616.867853] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2616.911725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2617.055178] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2627.061577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2627.089975] Lustre: Skipped 2 previous similar messages [ 2627.098709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2632.164987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2632.179696] Lustre: Skipped 2 previous similar messages [ 2632.415597] Lustre: server umount lustre-OST0000 complete [ 2637.285311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.291835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2637.302210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.413519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.427607] Lustre: Skipped 2 previous similar messages [ 2643.128491] Lustre: server umount lustre-MDT0000 complete [ 2647.521869] LustreError: 69778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.534993] LustreError: 69778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2647.937700] LustreError: 68758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801837 with bad export cookie 18183662212537034741 [ 2647.948144] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2647.950130] LustreError: 68758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2648.313150] Lustre: server umount lustre-MDT0001 complete [ 2663.038652] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2666.197570] Key type lgssc unregistered [ 2666.527481] LNet: 73646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2666.535550] LNetError: 73646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2666.554225] LNet: Removed LNI 192.168.202.108@tcp [ 2667.474303] Key type .llcrypt unregistered [ 2667.476711] Key type ._llcrypt unregistered [ 2693.689028] Key type ._llcrypt registered [ 2693.691590] Key type .llcrypt registered [ 2693.813198] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2695.281597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2695.355683] alg: No test for adler32 (adler32-zlib) [ 2696.528662] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 2696.877865] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2698.559508] Key type lgssc registered [ 2700.129771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2710.304491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2710.325482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2711.747333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2717.469660] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2722.272731] LustreError: 75054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.310693] LustreError: 75054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2727.396613] LustreError: 75055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.679340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2729.217032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2734.959974] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2744.150145] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2754.308444] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2766.111665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2766.735907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.028731] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2782.203317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2785.653826] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2792.579688] Lustre: Failing over lustre-MDT0000 [ 2792.752448] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 2792.907822] Lustre: server umount lustre-MDT0000 complete [ 2796.016705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2796.024890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2796.036706] LustreError: 75055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.541351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2797.554936] Lustre: Skipped 1 previous similar message [ 2802.659985] LustreError: 75997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2802.695691] LustreError: 75997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2805.616217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.890499] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2806.066917] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2806.224252] LustreError: 78144:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2806.238527] LustreError: 78144:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.108@tcp: cfg command failed: rc = -9 [ 2806.248411] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2806.248411] [ 2806.265077] LustreError: MGC192.168.202.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2806.286569] LustreError: 78132:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2806.300942] LustreError: 78132:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2806.321892] LustreError: 78132:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2806.328377] LustreError: 78132:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2806.371897] Lustre: server umount lustre-MDT0000 complete [ 2806.377408] LustreError: 78132:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2807.785690] LustreError: 75055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2807.828883] LustreError: 75055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2814.735119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.973545] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2814.981903] LustreError: 75047:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2815.338770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.400247] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2818.164464] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2820.621291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2820.666109] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2820.750285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2821.644257] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2835.945424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.948346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2835.958340] Lustre: Skipped 1 previous similar message [ 2835.979882] Lustre: Skipped 1 previous similar message [ 2837.728072] Lustre: server umount lustre-OST0000 complete [ 2846.180508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.186689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2846.200645] Lustre: Skipped 1 previous similar message [ 2846.201036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2846.224282] Lustre: Skipped 1 previous similar message [ 2847.857930] Lustre: server umount lustre-MDT0000 complete [ 2851.299542] LustreError: 75055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.327276] LustreError: 75055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2851.457759] LustreError: 75036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802040 with bad export cookie 4673299501024749242 [ 2851.468419] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2851.475234] LustreError: 75036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2851.997142] Lustre: server umount lustre-MDT0001 complete [ 2864.977315] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2868.576072] Key type lgssc unregistered [ 2868.923939] LNet: 79932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2868.934500] LNetError: 79932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2868.960371] LNet: Removed LNI 192.168.202.108@tcp [ 2870.154591] Key type .llcrypt unregistered [ 2870.156272] Key type ._llcrypt unregistered [ 2885.023188] Key type ._llcrypt registered [ 2885.024795] Key type .llcrypt registered [ 2885.159361] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2885.773430] Key type .llcrypt unregistered [ 2885.793652] Key type ._llcrypt unregistered [ 2913.949381] Key type ._llcrypt registered [ 2913.953135] Key type .llcrypt registered [ 2914.110702] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2915.935366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2916.013959] alg: No test for adler32 (adler32-zlib) [ 2917.089280] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 2917.464815] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2919.195070] Key type lgssc registered [ 2921.461885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2931.678473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2931.697288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.354510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2938.799607] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2942.956590] LustreError: 81828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.996354] LustreError: 81828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2948.067684] LustreError: 81827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2949.628647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.152698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2955.967315] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2964.761852] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2971.991743] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2980.312099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2980.688558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.323785] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2995.342733] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3003.262399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 3016.674577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3016.688870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3016.714868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3018.731367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3018.736903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.744886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3018.759448] Lustre: Skipped 1 previous similar message [ 3022.670784] Lustre: server umount lustre-OST0000 complete [ 3026.913500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3026.918114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.929935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3028.961533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.965739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3028.979964] Lustre: Skipped 1 previous similar message [ 3033.025914] Lustre: server umount lustre-MDT0000 complete [ 3034.080379] LustreError: 81827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3034.121676] LustreError: 81827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3037.163305] LustreError: 81809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802226 with bad export cookie 1576673893164822790 [ 3037.177514] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3037.622861] Lustre: server umount lustre-MDT0001 complete [ 3049.509459] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3053.237100] Key type lgssc unregistered [ 3053.508466] LNet: 85383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3053.516591] LNetError: 85383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3053.533925] LNet: Removed LNI 192.168.202.108@tcp [ 3054.755069] Key type .llcrypt unregistered [ 3054.760837] Key type ._llcrypt unregistered [ 3084.240352] Key type ._llcrypt registered [ 3084.244544] Key type .llcrypt registered [ 3084.425476] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3085.748109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3085.779442] alg: No test for adler32 (adler32-zlib) [ 3087.377343] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 3087.866714] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3089.615194] Key type lgssc registered [ 3091.389529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3101.523833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3101.535622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.235028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3109.047919] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3113.454495] LustreError: 86796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.491499] LustreError: 86796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3118.565346] LustreError: 86795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.744629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.186465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3124.413711] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3133.164714] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3139.536089] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3148.755316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.243681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3156.654936] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3165.597751] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3173.921563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 3184.098123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3184.109966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.124302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3185.121051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3185.129359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.140744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3187.178246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3187.188345] Lustre: Skipped 1 previous similar message [ 3188.972746] Lustre: server umount lustre-OST0000 complete [ 3195.878424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3195.892831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.933258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3199.730723] Lustre: server umount lustre-MDT0000 complete [ 3202.533543] LustreError: 86791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3202.581098] LustreError: 86791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3204.291690] LustreError: 86776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802393 with bad export cookie 8064072476460415257 [ 3204.310486] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3204.648438] Lustre: server umount lustre-MDT0001 complete [ 3219.232249] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3224.852978] Key type lgssc unregistered [ 3225.254789] LNet: 90352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3225.270963] LNetError: 90352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3226.284500] LNet: Removed LNI 192.168.202.108@tcp [ 3227.386464] Key type .llcrypt unregistered [ 3227.388656] Key type ._llcrypt unregistered [ 3257.882918] Key type ._llcrypt registered [ 3257.885706] Key type .llcrypt registered [ 3258.012418] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3259.365510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3259.394911] alg: No test for adler32 (adler32-zlib) [ 3260.778822] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 3261.163247] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3262.903230] Key type lgssc registered [ 3264.569516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3274.469246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3274.479782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3276.242985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3282.811587] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3286.529699] LustreError: 91763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3286.553577] LustreError: 91763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3291.638916] LustreError: 91764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.546863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.179946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3301.000775] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3311.550332] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3319.080748] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3327.109400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3327.496226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.809181] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3345.368280] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3345.977596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3361.250532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.256063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3361.287626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3366.374916] LustreError: 93587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.601320] Lustre: server umount lustre-OST0000 complete [ 3371.490994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3371.493573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3371.511649] Lustre: Skipped 2 previous similar messages [ 3371.519985] Lustre: Skipped 2 previous similar messages [ 3376.613251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3376.632807] Lustre: Skipped 1 previous similar message [ 3377.918203] Lustre: server umount lustre-MDT0000 complete [ 3381.733108] LustreError: 92491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3381.762149] LustreError: 92491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3382.784201] LustreError: 91745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802572 with bad export cookie 453745713092840425 [ 3382.785823] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3382.794810] LustreError: 91745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3383.540829] Lustre: server umount lustre-MDT0001 complete [ 3399.001726] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3403.583850] Key type lgssc unregistered [ 3403.975893] LNet: 95319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3403.993864] LNetError: 95319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3404.022725] LNet: Removed LNI 192.168.202.108@tcp [ 3405.370084] Key type .llcrypt unregistered [ 3405.374925] Key type ._llcrypt unregistered [ 3436.445662] Key type ._llcrypt registered [ 3436.448362] Key type .llcrypt registered [ 3436.595418] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3438.470752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3438.538649] alg: No test for adler32 (adler32-zlib) [ 3440.147782] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 3440.602289] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3442.432164] Key type lgssc registered [ 3444.399436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3454.171913] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3454.199863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3470.047414] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3475.231433] LustreError: 96703:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 3501.922368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3509.491931] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3519.694743] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3526.111558] Lustre: server umount lustre-OST0000 complete [ 3536.947376] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 13:38:45 (1788802725) [ 3545.063609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.717513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3551.199475] LustreError: 98342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3551.235107] LustreError: 98342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3551.782228] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3555.302463] LustreError: 98343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.425041] LustreError: 98342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3563.256714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.923225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3570.435234] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3579.188511] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3586.497455] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3595.462567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3595.948204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3603.961514] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3613.709907] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3615.426945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3630.057688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3630.063669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3630.076930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3631.078407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3631.081078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3631.101985] Lustre: Skipped 1 previous similar message [ 3633.216470] Lustre: server umount lustre-OST0000 complete [ 3640.802132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3640.807523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.816053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3645.081547] Lustre: server umount lustre-MDT0000 complete [ 3646.438814] LustreError: 100417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3646.464820] LustreError: 100417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3649.667863] LustreError: 99078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802838 with bad export cookie 16304313715541351463 [ 3649.675472] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3649.682743] LustreError: 99078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3650.034423] Lustre: server umount lustre-MDT0001 complete [ 3663.234378] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3666.810546] Key type lgssc unregistered [ 3667.223054] LNet: 102037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3667.227626] LNetError: 102037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3667.244982] LNet: Removed LNI 192.168.202.108@tcp [ 3668.206405] Key type .llcrypt unregistered [ 3668.208305] Key type ._llcrypt unregistered [ 3718.744608] Key type ._llcrypt registered [ 3718.750814] Key type .llcrypt registered [ 3718.866662] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3720.402040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3720.449786] alg: No test for adler32 (adler32-zlib) [ 3721.622436] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 3721.826738] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3723.543902] Key type lgssc registered [ 3725.304588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3735.474676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3735.502139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3737.207346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3744.034337] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3746.794288] LustreError: 104224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3746.814851] LustreError: 104224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3751.928488] LustreError: 104225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3756.472293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.150538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3763.244046] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3772.272136] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3778.792750] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3785.785216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3786.163485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3793.202576] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3803.164669] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3807.746498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3823.075612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3823.078568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.081600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.096809] Lustre: Skipped 1 previous similar message [ 3824.675769] Lustre: server umount lustre-OST0000 complete [ 3828.705641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3828.722266] LustreError: Skipped 1 previous similar message [ 3828.727221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.748492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3828.757780] Lustre: Skipped 1 previous similar message [ 3833.319094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3833.328567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3834.730705] Lustre: server umount lustre-MDT0000 complete [ 3838.434261] LustreError: 105509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3838.452796] LustreError: 105509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3840.324796] LustreError: 106824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803029 with bad export cookie 17570446993587504788 [ 3840.328443] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3840.341523] LustreError: 106824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3840.760823] Lustre: server umount lustre-MDT0001 complete [ 3854.123058] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3857.498847] Key type lgssc unregistered [ 3857.770912] LNet: 107777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3857.776137] LNetError: 107777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3857.796641] LNet: Removed LNI 192.168.202.108@tcp [ 3859.077714] Key type .llcrypt unregistered [ 3859.079778] Key type ._llcrypt unregistered [ 3880.965986] Key type ._llcrypt registered [ 3880.968915] Key type .llcrypt registered [ 3881.213029] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3883.269644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3883.349528] alg: No test for adler32 (adler32-zlib) [ 3885.241681] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 3885.700251] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3887.449101] Key type lgssc registered [ 3888.753941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3898.885863] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3898.915147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3914.655482] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3919.776354] LustreError: 109016:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 3946.463228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3953.641118] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3963.234713] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3970.873908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3971.300431] LustreError: MGC192.168.202.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3971.325671] LustreError: 110041:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3971.344992] LustreError: 110041:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3971.379124] LustreError: 110041:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3971.469675] Lustre: server umount lustre-MDT0000 complete [ 3971.471870] LustreError: 110041:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3990.495434] Lustre: 108352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788803163/real 1788803163] req@00000000cfc0b3f9 x1875695973174656/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788803179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3990.541817] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3990.570307] LustreError: 109028:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3990.998323] Lustre: server umount lustre-OST0000 complete [ 4003.066381] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 4016.918560] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 4059.547951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.765135] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 4059.800824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4059.824232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4059.932845] Lustre: lustre-MDT0000: new disk, initializing [ 4060.070890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4060.099598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4065.565407] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4078.999459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4079.152356] Lustre: 112843:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4079.213678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4079.217549] Lustre: Skipped 1 previous similar message [ 4079.285337] Lustre: lustre-MDT0001: new disk, initializing [ 4079.425822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4079.504517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4079.528337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4085.649228] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4098.048483] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4104.030672] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4111.886585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4112.237300] Lustre: lustre-OST0000: new disk, initializing [ 4112.243443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4112.252956] Lustre: 114813:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4112.372949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4120.741198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4120.755503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4120.851231] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4120.984659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4132.008479] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4137.842251] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4138.231079] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4143.661965] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4143.934888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4146.656056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4146.659323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.674618] LustreError: Skipped 1 previous similar message [ 4146.681944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4146.704052] Lustre: Skipped 1 previous similar message [ 4146.723646] Lustre: Skipped 1 previous similar message [ 4151.776770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4151.779379] Lustre: Skipped 1 previous similar message [ 4152.740945] Lustre: server umount lustre-OST0000 complete [ 4161.517467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4161.536397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4161.554565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4163.341849] Lustre: server umount lustre-MDT0000 complete [ 4167.593602] LustreError: 112834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803356 with bad export cookie 13034920826032806076 [ 4167.597710] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4167.617548] LustreError: 112834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4168.099603] Lustre: server umount lustre-MDT0001 complete [ 4178.907816] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 13:49:27 (1788803367) [ 4183.047854] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4201.725978] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 4212.728439] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 4253.576873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4254.064387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4254.116334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4254.189231] Lustre: lustre-MDT0000: new disk, initializing [ 4254.368908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.405546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4260.035156] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4272.073062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4272.269752] Lustre: 119717:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4272.415057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4272.426821] Lustre: Skipped 1 previous similar message [ 4272.568587] Lustre: lustre-MDT0001: new disk, initializing [ 4272.723045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4272.775731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4272.789604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4277.802994] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4288.864195] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4294.992988] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4301.635269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4301.993546] Lustre: lustre-OST0000: new disk, initializing [ 4302.002380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4302.012614] Lustre: 121689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4302.192187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4308.934910] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4309.593284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4309.614353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4309.723954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4319.735911] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4324.729519] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4325.000939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4329.844573] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4330.121707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4335.585226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4335.586572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4335.600799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4335.618170] Lustre: Skipped 2 previous similar messages [ 4335.635559] Lustre: Skipped 3 previous similar messages [ 4338.608353] Lustre: server umount lustre-OST0000 complete [ 4344.288741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4344.294788] LustreError: Skipped 1 previous similar message [ 4344.303140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4344.338830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4349.247561] Lustre: server umount lustre-MDT0000 complete [ 4350.951880] LustreError: 120894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.985018] LustreError: 120894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4353.526170] LustreError: 119709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803542 with bad export cookie 13034920826032806958 [ 4353.527874] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4353.546328] LustreError: 119709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4353.953791] Lustre: server umount lustre-MDT0001 complete [ 4364.490828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4365.082326] LustreError: 123904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4365.148583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4370.409623] LustreError: 123905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.454898] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4375.530412] LustreError: 123904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4380.645952] LustreError: 123905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4380.898290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.694652] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4394.746684] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4401.336302] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4408.377447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4414.924441] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4423.361805] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4433.768225] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4440.544251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4440.555501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.571903] Lustre: Skipped 1 previous similar message [ 4440.580649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4440.587684] Lustre: Skipped 2 previous similar messages [ 4444.611782] Lustre: server umount lustre-OST0000 complete [ 4451.809799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.839151] Lustre: Skipped 2 previous similar messages [ 4454.805887] Lustre: server umount lustre-MDT0000 complete [ 4456.931916] LustreError: 123901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4456.953549] LustreError: 123901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4458.575565] LustreError: 123886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803647 with bad export cookie 13034920826032807840 [ 4458.605388] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4458.956821] Lustre: server umount lustre-MDT0001 complete [ 4470.519237] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 4473.779265] Key type lgssc unregistered [ 4474.197801] LNet: 127531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4474.216591] LNetError: 127531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4474.254265] LNet: Removed LNI 192.168.202.108@tcp [ 4475.393940] Key type .llcrypt unregistered [ 4475.396796] Key type ._llcrypt unregistered [ 4495.485456] Key type ._llcrypt registered [ 4495.489355] Key type .llcrypt registered [ 4495.687700] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 4511.929574] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 4513.403509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4513.497134] alg: No test for adler32 (adler32-zlib) [ 4514.764308] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 4515.261109] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 4517.023213] Key type lgssc registered [ 4518.365059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4558.744683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4558.771664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.057607] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 4560.084662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4560.119794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4560.218704] Lustre: lustre-MDT0000: new disk, initializing [ 4560.293525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4560.314753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4564.685585] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4575.299223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4575.454210] Lustre: 130684:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4575.524349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4575.533774] Lustre: Skipped 1 previous similar message [ 4575.666510] Lustre: lustre-MDT0001: new disk, initializing [ 4575.798677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4575.851259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4575.868207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4582.691880] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4595.061600] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4602.763951] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4612.560833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4612.733223] Lustre: lustre-OST0000: new disk, initializing [ 4612.737433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4612.743074] Lustre: 132656:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4612.816303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4619.834944] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4619.912503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4619.928627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4620.118655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4631.432411] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4636.856656] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4637.126250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4643.318659] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4643.534215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4645.867754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4645.873559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4645.882464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4650.977630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4650.989289] Lustre: Skipped 2 previous similar messages [ 4651.974164] Lustre: server umount lustre-OST0000 complete [ 4657.632515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4657.643061] LustreError: Skipped 1 previous similar message [ 4657.650689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4657.671959] Lustre: Skipped 1 previous similar message [ 4657.680726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4661.229803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.235722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4661.256422] Lustre: Skipped 1 previous similar message [ 4663.064530] Lustre: server umount lustre-MDT0000 complete [ 4667.990637] LustreError: 130678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803857 with bad export cookie 17114894326902615779 [ 4668.015533] LustreError: 130678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4668.018124] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4668.566275] Lustre: server umount lustre-MDT0001 complete [ 4679.598613] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 13:57:47 (1788803867) [ 4686.572723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4687.216522] LustreError: 135068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.235397] LustreError: 135068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4687.288897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4692.244786] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4692.463522] LustreError: 135069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4697.575603] LustreError: 135068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.638585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.696483] LustreError: 135069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4703.078890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4709.121846] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4718.759619] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4726.048241] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4728.395579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.398812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4728.407701] Lustre: Skipped 1 previous similar message [ 4728.419949] Lustre: Skipped 1 previous similar message [ 4734.825505] Lustre: server umount lustre-MDT0000 complete [ 4738.528385] LustreError: 136003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.546716] LustreError: 136003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4738.869465] LustreError: 135803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803928 with bad export cookie 17114894326902616661 [ 4738.886530] LustreError: 135803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4738.891909] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4739.501161] Lustre: server umount lustre-MDT0001 complete [ 4749.318460] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 13:58:57 (1788803937) [ 4756.243067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4772.001469] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4777.122364] LustreError: 137485:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 4802.911509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4809.581627] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4817.543800] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4820.210046] Lustre: server umount lustre-OST0000 complete [ 4829.522679] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 14:00:17 (1788804017) [ 4836.553484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4836.987078] LustreError: 138933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4837.067787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.781669] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4853.040567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4860.051921] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4869.764208] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4876.958388] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4886.593752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4886.994094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.004792] Lustre: Skipped 1 previous similar message [ 4896.300411] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4906.638768] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4920.298016] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4920.308180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.330378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4920.333393] Lustre: Skipped 2 previous similar messages [ 4926.425463] Lustre: server umount lustre-MDT0000 complete [ 4927.969578] LustreError: 138929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4927.992616] LustreError: 138929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4930.792733] LustreError: 138913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804120 with bad export cookie 17114894326902617326 [ 4930.794995] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4930.822828] LustreError: 138913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4931.387171] Lustre: server umount lustre-MDT0001 complete [ 4946.880490] Lustre: server umount lustre-OST0000 complete [ 4957.267518] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 14:02:25 (1788804145) [ 4965.146556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4965.595104] LustreError: 142625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4965.708574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4971.863232] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4982.889117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4989.489681] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4997.992558] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5005.526950] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5012.771342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5020.589536] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5029.108624] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5034.744132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 5035.509195] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5035.771192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5040.826094] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5041.084475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5044.194808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5044.218432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5044.233275] Lustre: Skipped 2 previous similar messages [ 5044.237174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5044.252885] Lustre: Skipped 3 previous similar messages [ 5049.347242] Lustre: server umount lustre-OST0000 complete [ 5054.972178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5054.988106] LustreError: Skipped 1 previous similar message [ 5060.124037] Lustre: server umount lustre-MDT0000 complete [ 5060.586848] LustreError: 142621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5060.631379] LustreError: 142621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5065.404508] LustreError: 142605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804254 with bad export cookie 17114894326902618397 [ 5065.411610] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5065.762876] Lustre: server umount lustre-MDT0001 complete [ 5075.880679] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 14:04:23 (1788804263) [ 5083.655510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5099.429118] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5104.544674] LustreError: 146783:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 5130.273262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5130.276717] Lustre: Skipped 2 previous similar messages [ 5137.803960] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5146.833931] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5155.784875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.693959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5162.627218] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5174.362122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5180.725839] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5190.417966] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5197.132370] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5202.654196] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5202.918345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5207.603568] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5207.801916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5210.592930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5210.598702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5210.607568] Lustre: Skipped 3 previous similar messages [ 5210.612910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5210.617650] Lustre: Skipped 5 previous similar messages [ 5215.699653] Lustre: server umount lustre-MDT0000 complete [ 5216.224366] LustreError: 149122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5216.262098] LustreError: 149122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5220.217718] LustreError: 146789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804409 with bad export cookie 17114894326902619419 [ 5220.245240] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5220.710326] Lustre: server umount lustre-MDT0001 complete [ 5236.440203] Lustre: server umount lustre-OST0000 complete [ 5247.715523] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 14:07:15 (1788804435) [ 5257.047574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5272.800227] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5277.919848] LustreError: 150986:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 5311.831055] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5322.099948] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5331.196162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5331.897671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5337.403884] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5348.322250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5354.330890] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5363.861615] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5371.559581] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5380.081603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5380.459055] Lustre: lustre-OST0001: new disk, initializing [ 5380.465436] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5380.477615] Lustre: 153859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5388.328927] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5388.902616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5388.914505] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5389.028816] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5400.051685] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5406.130492] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5406.426650] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5412.235333] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5412.602929] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5414.892252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5414.911210] Lustre: Skipped 2 previous similar messages [ 5414.914665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5414.921937] Lustre: Skipped 4 previous similar messages [ 5429.215960] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5429.345633] Lustre: server umount lustre-OST0000 complete [ 5435.377295] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5440.817233] Lustre: server umount lustre-OST0001 complete [ 5452.220733] Lustre: server umount lustre-MDT0000 complete [ 5456.418025] LustreError: 153871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804645 with bad export cookie 17114894326902620371 [ 5456.426342] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5456.427801] LustreError: 153871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5456.789915] Lustre: server umount lustre-MDT0001 complete [ 5489.600693] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 14:11:17 (1788804677) [ 5491.377351] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5493.448661] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5495.341807] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 14:11:23 (1788804683) [ 5503.350091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.716227] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5503.780769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5504.014621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5504.024179] Lustre: Skipped 6 previous similar messages [ 5509.995243] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5521.834900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.926589] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5521.951935] Lustre: 157428:0:(mgs_llog.c:1462:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5528.129338] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5538.241387] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5546.075122] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5555.597739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5555.840066] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5563.409228] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5563.559404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5574.691812] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5581.284083] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5581.671941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5587.331519] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5587.687259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5594.081193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5594.086796] LustreError: Skipped 1 previous similar message [ 5594.089355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5594.098333] Lustre: Skipped 5 previous similar messages [ 5594.102787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5594.107361] Lustre: Skipped 12 previous similar messages [ 5604.319387] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5604.553386] Lustre: server umount lustre-OST0000 complete [ 5629.876266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5638.144238] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5649.229625] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5649.837919] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.865585] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5649.868291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.220707] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5657.440965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5664.366486] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5664.642422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5673.133468] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b950690c800.ost_server_uuid 50 [ 5674.928204] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b950690c800.ost_server_uuid in IDLE state after 0 sec [ 5675.531884] LustreError: 159926:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.8@tcp) returned error from blocking AST (req@00000000414c9ab5 x1875696633016832 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b3aa6f7e/0xed8451b633610a2d lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.8@tcp remote: 0xb0e458c70034aa83 expref: 5 pid: 157434 timeout: 5775 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5675.594901] LustreError: lustre-MDT0000: A client on nid 192.168.202.8@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5675.632151] LustreError: 157423:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.8@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b3aa6f7e/0xed8451b633610a2d lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.8@tcp remote: 0xb0e458c70034aa83 expref: 6 pid: 157434 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5682.154557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5682.162285] LustreError: Skipped 1 previous similar message [ 5687.464849] Lustre: server umount lustre-OST0000 complete [ 5699.045818] Lustre: server umount lustre-MDT0000 complete [ 5701.093271] LustreError: 157437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.148534] LustreError: 157437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 5703.929275] LustreError: 159287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804893 with bad export cookie 17114894326902621554 [ 5703.946097] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5703.948596] LustreError: 159287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5710.633224] Lustre: server umount lustre-MDT0001 complete [ 5724.664605] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 5728.435532] Key type lgssc unregistered [ 5728.835564] LNet: 163544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5728.854811] LNetError: 163544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5729.898215] LNet: Removed LNI 192.168.202.108@tcp [ 5730.763117] Key type .llcrypt unregistered [ 5730.764438] Key type ._llcrypt unregistered [ 5758.894762] Key type ._llcrypt registered [ 5758.897385] Key type .llcrypt registered [ 5759.047749] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 5760.292122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5760.603316] alg: No test for adler32 (adler32-zlib) [ 5761.984446] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 5762.441381] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 5764.175244] Key type lgssc registered [ 5765.546892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5774.965567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5774.998110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5776.546791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5781.973869] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5786.088411] LustreError: 164958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5786.119024] LustreError: 164958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5791.215317] LustreError: 164959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5791.679864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.229235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5797.863220] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5806.911804] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5813.957888] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5822.479208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5822.808194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5829.964677] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5839.291846] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5846.400865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5852.663651] Lustre: Failing over lustre-MDT0000 [ 5853.077346] Lustre: server umount lustre-MDT0000 complete [ 5853.668244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5853.688242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5853.729631] LustreError: 165686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.736832] LustreError: 164958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.740726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5856.753555] LustreError: 164958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5856.804888] Lustre: Skipped 1 previous similar message [ 5861.865674] LustreError: 164954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.885218] LustreError: 164954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5865.877790] LustreError: 164937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.8@tcp arrived at 1788805055 with bad export cookie 7838235202532496772 [ 5872.099799] LustreError: 164953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5872.125374] LustreError: 164953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5873.123726] Lustre: 164269:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788805046/real 1788805046] req@0000000009964985 x1875697940329984/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788805062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5873.156524] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5874.557328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.730508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5883.798536] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5888.634587] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5888.999926] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5889.024431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5891.379191] Lustre: lustre-MDT0000: Denying connection for new client 0913a43b-f6b8-4519-af83-52336cb99e19 (at 192.168.202.8@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5903.840776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5903.851244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.875199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5905.986348] Lustre: server umount lustre-OST0000 complete [ 5910.207971] LustreError: 168708:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5910.218511] Lustre: 168058:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5910.224797] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5910.238985] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5910.248555] LustreError: 168058:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b4682ca7 x1875697940350976/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5910.271767] LustreError: 168058:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5910.281557] LustreError: 168058:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5910.293550] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5910.659184] Lustre: server umount lustre-MDT0000 complete [ 5914.594644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5914.636206] Lustre: Skipped 1 previous similar message [ 5914.657504] LustreError: 164955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5914.678868] LustreError: 164955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5915.092878] LustreError: 164939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805104 with bad export cookie 7838235202532497101 [ 5915.109111] LustreError: 164939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5915.113294] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5917.622243] Lustre: server umount lustre-MDT0001 complete [ 5930.191663] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 5933.344530] Key type lgssc unregistered [ 5933.874634] LNet: 169461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5933.883657] LNetError: 169461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5934.949320] LNet: Removed LNI 192.168.202.108@tcp [ 5935.898980] Key type .llcrypt unregistered [ 5935.901613] Key type ._llcrypt unregistered [ 5962.910228] Key type ._llcrypt registered [ 5962.912183] Key type .llcrypt registered [ 5962.989951] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 5964.478850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5964.495040] alg: No test for adler32 (adler32-zlib) [ 5965.594997] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 5965.951390] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 5967.671746] Key type lgssc registered [ 5968.989688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5978.109069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5978.144886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.629274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5985.083662] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5988.835488] LustreError: 170874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5988.848890] LustreError: 170874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5993.970266] LustreError: 170875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5995.017152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5995.495878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6001.039973] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6008.815790] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6015.734753] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6023.499491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6023.802721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6024.823785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6031.113304] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6040.452465] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6050.799202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6050.814464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6050.823243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6050.835027] Lustre: Skipped 1 previous similar message [ 6050.847827] Lustre: Skipped 1 previous similar message [ 6054.220357] Lustre: server umount lustre-OST0000 complete [ 6061.026646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.029071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6061.040919] Lustre: Skipped 1 previous similar message [ 6061.063074] Lustre: Skipped 1 previous similar message [ 6065.055327] Lustre: server umount lustre-MDT0000 complete [ 6066.152944] LustreError: 170870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6066.167440] LustreError: 170870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6069.381194] LustreError: 170855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805258 with bad export cookie 10070203851516308642 [ 6069.390168] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6069.814974] Lustre: server umount lustre-MDT0001 complete [ 6083.343490] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 6086.538451] Key type lgssc unregistered [ 6086.948633] LNet: 174408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6086.969167] LNetError: 174408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6086.993278] LNet: Removed LNI 192.168.202.108@tcp [ 6088.051987] Key type .llcrypt unregistered [ 6088.060077] Key type ._llcrypt unregistered [ 6128.516484] Key type ._llcrypt registered [ 6128.518965] Key type .llcrypt registered [ 6128.634305] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 6129.652639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6129.785135] alg: No test for adler32 (adler32-zlib) [ 6131.229750] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 6131.571114] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 6133.255173] Key type lgssc registered [ 6134.526851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6142.961328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6142.988989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.423218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6149.364987] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6153.706054] LustreError: 176105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.754362] LustreError: 176105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6158.266346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6158.594774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6163.522466] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6172.193514] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6177.822578] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6185.593265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6185.944384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6189.053905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6192.066349] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6200.132410] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6214.130774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6214.134722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6214.148580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6218.721155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6218.730713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.732538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6220.094592] Lustre: server umount lustre-OST0000 complete [ 6225.376850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6225.387593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6225.404951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6225.411884] Lustre: Skipped 1 previous similar message [ 6228.962581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.970623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6229.009490] Lustre: Skipped 1 previous similar message [ 6231.005291] Lustre: server umount lustre-MDT0000 complete [ 6234.080897] LustreError: 176833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6234.112265] LustreError: 176833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6235.332076] LustreError: 176087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805424 with bad export cookie 2527496277792152001 [ 6235.339178] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6235.342274] LustreError: 176087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6235.803252] Lustre: server umount lustre-MDT0001 complete [ 6248.730735] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 6251.657698] Key type lgssc unregistered [ 6251.954807] LNet: 179657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6251.961665] LNetError: 179657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6251.980303] LNet: Removed LNI 192.168.202.108@tcp [ 6252.602528] Key type .llcrypt unregistered [ 6252.604486] Key type ._llcrypt unregistered [ 6278.592264] Key type ._llcrypt registered [ 6278.594522] Key type .llcrypt registered [ 6278.719762] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 6280.007907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6280.191810] alg: No test for adler32 (adler32-zlib) [ 6281.205616] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 6281.415920] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 6283.079545] Key type lgssc registered [ 6284.345449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6294.521781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6294.541570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6295.860363] Lustre: *** cfs_fail_loc=135, val=0*** [ 6295.977895] LustreError: 181177:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6295.987172] LustreError: 181177:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.108@tcp: cfg command failed: rc = -2 [ 6295.994115] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6295.994115] [ 6296.008945] LustreError: MGC192.168.202.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6296.041532] LustreError: 181138:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6296.048957] LustreError: 181138:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 6296.059845] LustreError: 181138:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 6296.073128] LustreError: 181138:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6296.148677] Lustre: server umount lustre-MDT0000 complete [ 6296.152637] LustreError: 181138:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6303.519637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6303.811622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6308.150919] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6313.958550] LustreError: 181547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.979851] LustreError: 181547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6316.460544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.769917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6321.292712] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6329.259479] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6334.611043] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6336.613647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6336.616960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6336.625674] Lustre: Skipped 1 previous similar message [ 6336.644764] Lustre: Skipped 1 previous similar message [ 6341.729729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6343.086330] Lustre: server umount lustre-MDT0000 complete [ 6346.201834] LustreError: 181529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805535 with bad export cookie 17799717571161422988 [ 6346.204837] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6346.217392] LustreError: 181529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6346.531797] Lustre: server umount lustre-MDT0001 complete [ 6357.277193] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 6360.236237] Key type lgssc unregistered [ 6360.574558] LNet: 183865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6360.582649] LNetError: 183865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6360.597680] LNet: Removed LNI 192.168.202.108@tcp [ 6361.495942] Key type .llcrypt unregistered [ 6361.498659] Key type ._llcrypt unregistered [ 6386.699204] Key type ._llcrypt registered [ 6386.701403] Key type .llcrypt registered [ 6386.876426] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 6387.174445] Key type .llcrypt unregistered [ 6387.175956] Key type ._llcrypt unregistered [ 6405.471489] Key type ._llcrypt registered [ 6405.474984] Key type .llcrypt registered [ 6405.645432] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 6407.003331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6407.097716] alg: No test for adler32 (adler32-zlib) [ 6408.120385] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 6408.347138] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 6410.057975] Key type lgssc registered [ 6411.124404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6419.654146] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6419.670797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6435.167420] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6440.287353] LustreError: 185991:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 6466.975212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6473.770437] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6482.580199] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6489.489820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.583382] LustreError: 185326:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002ed58cf7 x1875698617682176/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6489.960653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6490.023653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6494.830648] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6495.203028] LustreError: 186016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6495.226171] LustreError: 186016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6500.326769] LustreError: 186017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6501.351165] LustreError: 186016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6503.396448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6503.848778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6508.870946] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6516.896657] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6522.399844] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6525.418449] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6539.746692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6539.751039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6539.778311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6539.780917] Lustre: Skipped 1 previous similar message [ 6542.919412] Lustre: server umount lustre-OST0000 complete [ 6549.989741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6549.994773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6550.002259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6550.022580] Lustre: Skipped 2 previous similar messages [ 6550.047626] Lustre: Skipped 1 previous similar message [ 6555.107456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6560.224416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6560.230739] Lustre: Skipped 2 previous similar messages [ 6561.247354] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6561.528657] Lustre: server umount lustre-MDT0000 complete [ 6565.214780] LustreError: 185998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805754 with bad export cookie 12418746595809882601 [ 6565.216989] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6565.226386] LustreError: 185998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6565.498689] Lustre: server umount lustre-MDT0001 complete [ 6577.077970] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 6580.254197] Key type lgssc unregistered [ 6580.657659] LNet: 189958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6580.671166] LNetError: 189958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6580.717432] LNet: Removed LNI 192.168.202.108@tcp [ 6581.614407] Key type .llcrypt unregistered [ 6581.617667] Key type ._llcrypt unregistered [ 6603.764542] Key type ._llcrypt registered [ 6603.767789] Key type .llcrypt registered [ 6603.905899] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 6605.131625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6605.186304] alg: No test for adler32 (adler32-zlib) [ 6606.264880] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 6606.537976] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 6608.271937] Key type lgssc registered [ 6609.299431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6617.801522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6617.851980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.334985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6624.453203] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6628.838851] LustreError: 191347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6628.861201] LustreError: 191347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6633.964164] LustreError: 191346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6634.082451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6634.544547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6639.557140] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6648.265976] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6653.470981] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6660.407891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6660.780926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6664.879615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6667.383485] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6674.232838] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6679.707816] Lustre: Failing over lustre-MDT0000 [ 6679.927086] Lustre: server umount lustre-MDT0000 complete [ 6680.545514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6680.552490] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.569502] LustreError: 192076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6680.577899] Lustre: Skipped 1 previous similar message [ 6686.690646] LustreError: 192076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6686.709934] LustreError: 192076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6691.809142] LustreError: 192332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6691.824036] LustreError: 192332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6695.903310] Lustre: 190657:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788805869/real 1788805869] req@00000000320e1089 x1875698825320576/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788805885 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6695.930363] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6698.475543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6701.027776] LustreError: 191343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6701.044272] LustreError: 191343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6706.452156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6706.515599] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6706.968452] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6710.373644] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6711.802609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6711.836585] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6711.889579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6714.780613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6726.372156] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6741.994077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6742.005295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6742.025097] Lustre: Skipped 1 previous similar message [ 6742.033564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6743.991341] Lustre: server umount lustre-OST0000 complete [ 6752.752470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.767597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6752.770424] Lustre: Skipped 2 previous similar messages [ 6752.770783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6752.807321] Lustre: Skipped 3 previous similar messages [ 6754.490310] Lustre: server umount lustre-MDT0000 complete [ 6757.856948] LustreError: 192076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6757.871592] LustreError: 192076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6758.584740] LustreError: 191326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805947 with bad export cookie 4487795640836934559 [ 6758.585544] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6758.598372] LustreError: 191326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6758.994974] Lustre: server umount lustre-MDT0001 complete [ 6770.273969] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 6772.912167] Key type lgssc unregistered [ 6773.351622] LNet: 196351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6773.363865] LNetError: 196351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6773.395440] LNet: Removed LNI 192.168.202.108@tcp [ 6774.276996] Key type .llcrypt unregistered [ 6774.280666] Key type ._llcrypt unregistered [ 6798.120346] Key type ._llcrypt registered [ 6798.128733] Key type .llcrypt registered [ 6798.237414] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 6799.416919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6799.501707] alg: No test for adler32 (adler32-zlib) [ 6800.625788] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 6800.866122] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 6802.568083] Key type lgssc registered [ 6803.525448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6811.056655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6811.079366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.470434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6816.815623] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6821.867760] LustreError: 197759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6821.886769] LustreError: 197759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6825.306267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6825.818499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6830.224881] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6837.596107] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6843.931239] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6850.453405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6850.759666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6856.754953] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6856.940733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6864.637525] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6868.688480] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6875.523595] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6888.464683] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6893.024745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6893.037488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6893.044063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6896.097384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6896.099622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6896.104956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6896.123793] Lustre: Skipped 1 previous similar message [ 6899.099258] Lustre: server umount lustre-OST0000 complete [ 6902.753242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6902.758652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6902.786841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6906.344766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6906.345756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6908.811265] Lustre: server umount lustre-MDT0000 complete [ 6911.459543] LustreError: 197759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6911.481544] LustreError: 197759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6912.460984] LustreError: 200549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806101 with bad export cookie 4856516584226415744 [ 6912.463446] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6912.468858] LustreError: 200549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6912.817852] Lustre: server umount lustre-MDT0001 complete [ 6923.554816] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 6926.397257] Key type lgssc unregistered [ 6926.747565] LNet: 201603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6926.758432] LNetError: 201603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6926.773634] LNet: Removed LNI 192.168.202.108@tcp [ 6927.795897] Key type .llcrypt unregistered [ 6927.798968] Key type ._llcrypt unregistered [ 6945.961198] Key type ._llcrypt registered [ 6945.963833] Key type .llcrypt registered [ 6946.092189] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 6946.851263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6946.890284] alg: No test for adler32 (adler32-zlib) [ 6947.985956] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 6948.246861] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 6949.879221] Key type lgssc registered [ 6950.846685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6958.110230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6958.124121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6959.430122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6963.271867] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6968.808114] LustreError: 202864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6968.822841] LustreError: 202864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6971.305958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6971.702042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6975.244705] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6980.655694] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6985.852490] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6991.704854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6991.949538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6996.655852] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7002.578850] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7004.282946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 7012.229960] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 7023.580390] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7032.314918] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 7042.322691] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 7055.842039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7055.845511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7055.848185] Lustre: Skipped 1 previous similar message [ 7055.848567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7055.855014] Lustre: Skipped 1 previous similar message [ 7060.968656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7060.987062] Lustre: Skipped 1 previous similar message [ 7066.082316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7068.127392] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7068.271874] Lustre: server umount lustre-OST0000 complete [ 7074.272272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7074.277403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.286483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7074.290863] Lustre: Skipped 1 previous similar message [ 7076.333626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.448812] Lustre: server umount lustre-MDT0000 complete [ 7080.705847] LustreError: 205542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806270 with bad export cookie 9359345115144010018 [ 7080.709903] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7080.714044] LustreError: 205542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7080.956156] Lustre: server umount lustre-MDT0001 complete [ 7089.390285] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 7092.198831] Key type lgssc unregistered [ 7092.552809] LNet: 208072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7092.561238] LNetError: 208072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7092.594369] LNet: Removed LNI 192.168.202.108@tcp [ 7093.375278] Key type .llcrypt unregistered [ 7093.378465] Key type ._llcrypt unregistered [ 7111.179842] Key type ._llcrypt registered [ 7111.181664] Key type .llcrypt registered [ 7111.287365] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 7112.099914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7112.168387] alg: No test for adler32 (adler32-zlib) [ 7113.227354] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 7113.451737] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 7115.111244] Key type lgssc registered [ 7116.194124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7122.627659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7122.650186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7124.007129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7127.579753] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7134.179935] LustreError: 209460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7134.197804] LustreError: 209460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7134.368962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.623427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7138.063516] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7143.447798] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7147.919943] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7152.965813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7153.199912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7157.498587] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7163.395770] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7169.578928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7184.864843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7184.882710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7184.887374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7188.818798] Lustre: server umount lustre-OST0000 complete [ 7195.104573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7195.119833] Lustre: Skipped 1 previous similar message [ 7195.125253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7195.138271] Lustre: Skipped 2 previous similar messages [ 7197.739755] Lustre: server umount lustre-MDT0000 complete [ 7200.225324] LustreError: 209460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7200.238580] LustreError: 209460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7200.507928] LustreError: 209440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806389 with bad export cookie 7476223761334632594 [ 7200.513633] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7200.515888] LustreError: 209440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7200.776929] Lustre: server umount lustre-MDT0001 complete [ 7208.291889] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 7210.136800] Key type lgssc unregistered [ 7210.322989] LNet: 213163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7210.329827] LNetError: 213163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7210.345844] LNet: Removed LNI 192.168.202.108@tcp [ 7210.846281] Key type .llcrypt unregistered [ 7210.848446] Key type ._llcrypt unregistered [ 7224.860637] Key type ._llcrypt registered [ 7224.863597] Key type .llcrypt registered [ 7224.956733] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 7225.792430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7225.859250] alg: No test for adler32 (adler32-zlib) [ 7226.807490] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 7226.980904] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 7228.623178] Key type lgssc registered [ 7229.440219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7234.933832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7234.946531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7236.295859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7239.614381] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7245.800059] LustreError: 214402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7245.819634] LustreError: 214402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7246.198115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.436556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7249.662911] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7255.091266] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7259.561911] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7264.599796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7264.950742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7269.874994] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7275.567690] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7281.321213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7291.872706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7291.874494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7291.880174] LustreError: Skipped 1 previous similar message [ 7291.895062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7296.995364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7297.004989] Lustre: Skipped 1 previous similar message [ 7297.917489] Lustre: server umount lustre-OST0000 complete [ 7302.115891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7302.119598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7302.131252] Lustre: Skipped 2 previous similar messages [ 7302.138432] Lustre: Skipped 2 previous similar messages [ 7306.988344] Lustre: server umount lustre-MDT0000 complete [ 7307.233858] LustreError: 214401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7307.242477] LustreError: 214401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7310.165992] LustreError: 214382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806499 with bad export cookie 221987895957228742 [ 7310.166527] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7310.176786] LustreError: 214382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7310.473857] Lustre: server umount lustre-MDT0001 complete [ 7318.494509] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 7320.387538] Key type lgssc unregistered [ 7320.608688] LNet: 218097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7320.614342] LNetError: 218097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7320.627294] LNet: Removed LNI 192.168.202.108@tcp [ 7321.287453] Key type .llcrypt unregistered [ 7321.289695] Key type ._llcrypt unregistered [ 7334.491658] Key type ._llcrypt registered [ 7334.493729] Key type .llcrypt registered [ 7334.577727] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 7335.271098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7335.347454] alg: No test for adler32 (adler32-zlib) [ 7336.278862] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 7336.454123] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 7338.119226] Key type lgssc registered [ 7338.974732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7344.932810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7344.945407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7346.221307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7349.471686] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7356.072112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7356.330186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7359.812278] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7365.551498] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7369.633501] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7374.121449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7374.378093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7378.790085] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7384.896756] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7391.792151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7402.292927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7402.340300] LustreError: lustre-OST0001: New target registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new target registrations. [ 7402.354643] LustreError: lustre-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7402.371771] LustreError: 222185:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 7402.383519] LustreError: 222185:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0001 [ 7402.388284] LustreError: 222185:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0001 not registered [ 7402.399187] Lustre: server umount lustre-OST0001 complete [ 7402.402618] LustreError: 222185:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7404.991194] Lustre: DEBUG MARKER: conf-sanity test_29: @@@@@@ FAIL: Unable to start OST2 [ 7410.128903] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 14:43:19 (1788806599) [ 7422.002362] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7431.568163] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7436.715032] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7447.079930] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7456.595918] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7471.154552] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7471.158782] Lustre: Skipped 1 previous similar message [ 7491.610049] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7491.613204] Lustre: Skipped 2 previous similar messages [ 7525.797197] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7525.804373] Lustre: Skipped 4 previous similar messages [ 7581.151960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7581.155325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7581.161805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7581.164062] Lustre: Skipped 1 previous similar message [ 7585.911461] Lustre: server umount lustre-OST0000 complete [ 7591.392724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.396134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7591.402919] Lustre: Skipped 1 previous similar message [ 7591.405051] Lustre: Skipped 1 previous similar message [ 7594.325054] Lustre: server umount lustre-MDT0000 complete [ 7596.481554] LustreError: 223427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806785 with bad export cookie 10252937441965585619 [ 7596.483501] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7596.492534] LustreError: 223427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7596.675558] Lustre: server umount lustre-MDT0001 complete [ 7602.459812] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 7603.996256] Key type lgssc unregistered [ 7604.142756] LNet: 225587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7604.146575] LNetError: 225587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7604.160709] LNet: Removed LNI 192.168.202.108@tcp [ 7604.635790] Key type .llcrypt unregistered [ 7604.637609] Key type ._llcrypt unregistered [ 7619.097544] Key type ._llcrypt registered [ 7619.099474] Key type .llcrypt registered [ 7619.176682] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 7619.948289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7619.990709] alg: No test for adler32 (adler32-zlib) [ 7620.917312] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 7621.077231] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 7622.687270] Key type lgssc registered [ 7623.371736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7627.750484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7627.759236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7629.014612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.575919] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7636.350490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7636.593237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7638.902684] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7643.278985] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7646.261556] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7650.066530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7650.282853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7653.462456] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7654.396757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7657.514496] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7669.728440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7669.733259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7669.741912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7671.782045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7671.787349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7671.802902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7675.323361] Lustre: server umount lustre-OST0000 complete [ 7682.016546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.021939] Lustre: Skipped 1 previous similar message [ 7682.022416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7682.033041] Lustre: Skipped 2 previous similar messages [ 7683.850748] Lustre: server umount lustre-MDT0000 complete [ 7686.046237] LustreError: 226955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806875 with bad export cookie 7474695933966396992 [ 7686.050378] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7686.055779] LustreError: 226955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7686.239944] Lustre: server umount lustre-MDT0001 complete [ 7692.469396] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 7694.294475] Key type lgssc unregistered [ 7694.495704] LNet: 230659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7694.502500] LNetError: 230659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7694.511616] LNet: Removed LNI 192.168.202.108@tcp [ 7695.041403] Key type .llcrypt unregistered [ 7695.044451] Key type ._llcrypt unregistered [ 7707.530614] Key type ._llcrypt registered [ 7707.531943] Key type .llcrypt registered [ 7707.591485] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 7707.999428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7708.059725] alg: No test for adler32 (adler32-zlib) [ 7708.961809] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 7709.074953] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 7710.681958] Key type lgssc registered [ 7711.214093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7985.098760] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 7988.607654] Key type lgssc unregistered [ 7989.084240] LNet: 232656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7989.103488] LNetError: 232656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7989.137974] LNet: Removed LNI 192.168.202.108@tcp [ 7990.816911] Key type .llcrypt unregistered [ 7990.822604] Key type ._llcrypt unregistered [ 8040.251740] Key type ._llcrypt registered [ 8040.254988] Key type .llcrypt registered [ 8040.340671] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 8041.651676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8041.677710] alg: No test for adler32 (adler32-zlib) [ 8043.194563] Lustre: Lustre: Build Version: 2.17.58_2_gefa5002 [ 8043.357814] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 8045.031334] Key type lgssc registered [ 8047.267406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8055.764653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8055.800894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8057.164129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8062.258336] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8067.554625] LustreError: 234138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8067.573411] LustreError: 234138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8072.332674] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8072.684536] LustreError: 234139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8076.718153] LustreError: 234167:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 8076.733182] LustreError: 234167:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 8076.741034] LustreError: 234167:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 8077.086646] Lustre: server umount lustre-MDT0000 complete [ 8091.083073] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 8101.329319] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 8132.190967] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7799 sec ========== 14:55:20 (1788807320) [ 8133.891595] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:55:22 (1788807322) === [ 8137.780099] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:55:26 (1788807326) === [ 8173.887573] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 8176.351392] Key type lgssc unregistered [ 8176.651888] LNet: 239400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8176.661922] LNetError: 239400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8176.675540] LNet: Removed LNI 192.168.202.108@tcp [ 8177.518138] Key type .llcrypt unregistered [ 8177.520248] Key type ._llcrypt unregistered