[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 944556575 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002371] x2apic enabled [ 0.003017] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011151] LSM: Security Framework initializing [ 0.013062] Yama: becoming mindful. [ 0.014049] SELinux: Initializing. [ 0.015086] *** VALIDATE selinux *** [ 0.029223] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.035021] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.036153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.038111] *** VALIDATE tmpfs *** [ 0.040381] *** VALIDATE proc *** [ 0.041253] *** VALIDATE cgroup *** [ 0.042015] *** VALIDATE cgroup2 *** [ 0.043283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.044168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.046006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.047031] Spectre V2 : User space: Vulnerable [ 0.049008] Speculative Store Bypass: Vulnerable [ 0.054341] debug: unmapping init [mem 0xffffffffbc659000-0xffffffffbc660fff] [ 0.057000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.058762] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.059028] ... version: 2 [ 0.060019] ... bit width: 48 [ 0.061021] ... generic registers: 4 [ 0.062097] ... value mask: 0000ffffffffffff [ 0.063019] ... max period: 00007fffffffffff [ 0.064022] ... fixed-purpose events: 3 [ 0.066000] ... event mask: 000000070000000f [ 0.067158] rcu: Hierarchical SRCU implementation. [ 0.070719] smp: Bringing up secondary CPUs ... [ 0.071622] x86: Booting SMP configuration: [ 0.072037] .... node #0, CPUs: #1 #2 #3 [ 0.086883] smp: Brought up 1 node, 4 CPUs [ 0.088023] smpboot: Max logical packages: 1 [ 0.089019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146189] node 0 deferred pages initialised in 53ms [ 0.152110] devtmpfs: initialized [ 0.153366] x86/mm: Memory block size: 128MB [ 0.156581] gcov: version magic: 0x41383552 [ 0.158411] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159174] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160338] pinctrl core: initialized pinctrl subsystem [ 0.161260] [ 0.161820] ************************************************************* [ 0.162014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163016] ** ** [ 0.164017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165013] ** ** [ 0.166029] ** This means that this kernel is built to expose internal ** [ 0.167014] ** IOMMU data structures, which may compromise security on ** [ 0.168012] ** your system. ** [ 0.169012] ** ** [ 0.170013] ** If you see this message and you are not debugging the ** [ 0.171018] ** kernel, report this immediately to your vendor! ** [ 0.172012] ** ** [ 0.173016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174016] ************************************************************* [ 0.175902] NET: Registered protocol family 16 [ 0.176944] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177239] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178123] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179588] cpuidle: using governor menu [ 0.181419] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184484] PCI: Using configuration type 1 for base access [ 0.194129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203337] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.205034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209419] cryptd: max_cpu_qlen set to 1000 [ 0.210631] ACPI: Added _OSI(Module Device) [ 0.212018] ACPI: Added _OSI(Processor Device) [ 0.214011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.216011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.221524] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.230375] ACPI: Interpreter enabled [ 0.231069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.232016] ACPI: Using IOAPIC for interrupt routing [ 0.234065] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235418] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253177] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.254045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.256084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.260236] acpiphp: Slot [2] registered [ 0.261071] acpiphp: Slot [5] registered [ 0.262104] acpiphp: Slot [6] registered [ 0.263163] acpiphp: Slot [7] registered [ 0.264088] acpiphp: Slot [8] registered [ 0.266056] acpiphp: Slot [9] registered [ 0.267128] acpiphp: Slot [10] registered [ 0.269100] acpiphp: Slot [3] registered [ 0.270154] acpiphp: Slot [4] registered [ 0.271147] acpiphp: Slot [11] registered [ 0.272071] acpiphp: Slot [12] registered [ 0.273144] acpiphp: Slot [13] registered [ 0.274086] acpiphp: Slot [14] registered [ 0.275142] acpiphp: Slot [15] registered [ 0.276117] acpiphp: Slot [16] registered [ 0.277114] acpiphp: Slot [17] registered [ 0.278154] acpiphp: Slot [18] registered [ 0.279139] acpiphp: Slot [19] registered [ 0.280117] acpiphp: Slot [20] registered [ 0.281531] acpiphp: Slot [21] registered [ 0.282314] acpiphp: Slot [22] registered [ 0.283135] acpiphp: Slot [23] registered [ 0.284137] acpiphp: Slot [24] registered [ 0.285125] acpiphp: Slot [25] registered [ 0.286142] acpiphp: Slot [26] registered [ 0.287909] acpiphp: Slot [27] registered [ 0.288123] acpiphp: Slot [28] registered [ 0.289141] acpiphp: Slot [29] registered [ 0.290145] acpiphp: Slot [30] registered [ 0.291151] acpiphp: Slot [31] registered [ 0.292071] PCI host bridge to bus 0000:00 [ 0.293032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.294038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.295026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.297026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.298027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.299374] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.301311] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.303962] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.309906] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.312887] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.313025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.314020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.315022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.317071] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.319249] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.320045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.322106] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.325014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.333016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.336019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.342354] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.345016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.348026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.355036] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.361572] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.364024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.367018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.374015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.380591] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.383015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.386014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.395033] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.405047] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.408027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.411016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.419033] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.427818] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.431018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.434018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.441018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.447154] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.450018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.453020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.460016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.468054] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.469351] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.470423] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.471525] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.472187] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.537058] iommu: Default domain type: Passthrough [ 0.545489] SCSI subsystem initialized [ 0.557183] ACPI: bus type USB registered [ 0.559149] usbcore: registered new interface driver usbfs [ 0.566094] usbcore: registered new interface driver hub [ 0.578096] usbcore: registered new device driver usb [ 0.588224] pps_core: LinuxPPS API ver. 1 registered [ 0.598022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.607066] PTP clock support registered [ 0.610102] EDAC MC: Ver: 3.0.0 [ 0.611266] PCI: Using ACPI for IRQ routing [ 0.613227] NetLabel: Initializing [ 0.614016] NetLabel: domain hash size = 128 [ 0.615012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.616085] NetLabel: unlabeled traffic allowed by default [ 0.617136] vgaarb: loaded [ 0.618353] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.619012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.626418] clocksource: Switched to clocksource kvm-clock [ 1.121733] VFS: Disk quotas dquot_6.6.0 [ 1.123428] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.126133] *** VALIDATE ramfs *** [ 1.127445] *** VALIDATE hugetlbfs *** [ 1.129118] pnp: PnP ACPI init [ 1.131744] pnp: PnP ACPI: found 6 devices [ 1.148039] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.150853] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.153185] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.155521] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.158093] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.160695] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.163230] NET: Registered protocol family 2 [ 1.165677] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.170440] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.174098] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.179478] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.182924] TCP: Hash tables configured (established 65536 bind 65536) [ 1.185444] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.188232] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.191286] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.194661] NET: Registered protocol family 1 [ 1.197516] RPC: Registered named UNIX socket transport module. [ 1.202898] RPC: Registered udp transport module. [ 1.204716] RPC: Registered tcp transport module. [ 1.206266] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.211669] NET: Registered protocol family 44 [ 1.213253] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.216637] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.221535] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.225932] PCI: CLS 0 bytes, default 64 [ 1.228323] Unpacking initramfs... [ 4.195977] debug: unmapping init [mem 0xffff99533cc54000-0xffff99533ffbffff] [ 4.200269] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.202531] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.205160] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.780069] hrtimer: interrupt took 12744889 ns [ 6.382497] Initialise system trusted keyrings [ 6.385493] Key type blacklist registered [ 6.387841] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.402601] zbud: loaded [ 6.408220] *** VALIDATE nfs *** [ 6.409587] *** VALIDATE nfs4 *** [ 6.412057] pstore: using deflate compression [ 6.421414] Platform Keyring initialized [ 6.805574] NET: Registered protocol family 38 [ 6.810272] Key type asymmetric registered [ 6.813994] Asymmetric key parser 'x509' registered [ 6.818159] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.843691] io scheduler mq-deadline registered [ 6.847820] io scheduler kyber registered [ 6.852286] io scheduler bfq registered [ 6.856431] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.863397] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.871485] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.878613] ACPI: Power Button [PWRF] [ 6.892124] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.023480] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.069750] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.088764] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.123233] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.168028] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.209283] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.214654] Non-volatile memory driver v1.3 [ 7.216504] Linux agpgart interface v0.103 [ 7.247699] virtio_blk virtio1: [vda] 134632 512-byte logical blocks (68.9 MB/65.7 MiB) [ 7.251064] vda: detected capacity change from 0 to 68931584 [ 7.267313] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.270243] vdb: detected capacity change from 0 to 1073741824 [ 7.292618] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.296026] vdc: detected capacity change from 0 to 2621440000 [ 7.376852] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.429389] vdd: detected capacity change from 0 to 2621440000 [ 7.520868] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.530134] vde: detected capacity change from 0 to 4294967296 [ 7.596774] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.602449] vdf: detected capacity change from 0 to 4294967296 [ 7.629793] libphy: Fixed MDIO Bus: probed [ 7.649322] usbcore: registered new interface driver usbserial_generic [ 7.651123] usbserial: USB Serial support registered for generic [ 7.658026] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.667970] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.673934] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.679755] mousedev: PS/2 mouse device common for all mice [ 7.690899] rtc_cmos 00:05: RTC can wake from S4 [ 7.699045] rtc_cmos 00:05: registered as rtc0 [ 7.700468] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.701898] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.701963] intel_pstate: CPU model not supported [ 7.705741] hid: raw HID events driver (C) Jiri Kosina [ 7.715881] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.716635] usbcore: registered new interface driver usbhid [ 7.722903] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.729209] usbhid: USB HID core driver [ 7.731116] drop_monitor: Initializing network drop monitor service [ 7.733265] Initializing XFRM netlink socket [ 7.735493] NET: Registered protocol family 10 [ 7.738582] Segment Routing with IPv6 [ 7.741630] NET: Registered protocol family 17 [ 7.743739] mpls_gso: MPLS GSO support [ 7.748244] RAS: Correctable Errors collector initialized. [ 7.750157] AVX version of gcm_enc/dec engaged. [ 7.751907] AES CTR mode by8 optimization enabled [ 7.893163] sched_clock: Marking stable (7893023157, 0)->(11284609071, -3391585914) [ 7.903425] registered taskstats version 1 [ 7.905851] Loading compiled-in X.509 certificates [ 7.908114] zswap: loaded using pool lzo/zbud [ 7.997428] Key type big_key registered [ 8.008844] Key type encrypted registered [ 8.010354] ima: No TPM chip found, activating TPM-bypass! [ 8.011884] ima: Allocated hash algorithm: sha1 [ 8.013414] ima: No architecture policies found [ 8.015263] evm: Initialising EVM extended attributes: [ 8.017686] evm: security.selinux [ 8.018994] evm: security.ima [ 8.020147] evm: security.capability [ 8.021532] evm: HMAC attrs: 0x1 [ 8.024449] rtc_cmos 00:05: setting system clock to 2026-05-04 15:52:05 UTC (1777909925) [ 8.036787] debug: unmapping init [mem 0xffffffffbd603000-0xffffffffbd7fffff] [ 8.042747] debug: unmapping init [mem 0xffffffffbc382000-0xffffffffbc658fff] [ 8.051987] Write protecting the kernel read-only data: 28672k [ 8.067956] debug: unmapping init [mem 0xffffffffbaa03000-0xffffffffbabfffff] [ 8.071065] debug: unmapping init [mem 0xffffffffbb314000-0xffffffffbb3fffff] [ 8.145870] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.385451] systemd[1]: Detected virtualization kvm. [ 8.388195] systemd[1]: Detected architecture x86-64. [ 8.390765] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.509151] systemd[1]: No hostname configured. [ 8.514045] systemd[1]: Set hostname to . [ 8.517709] random: systemd: uninitialized urandom read (16 bytes read) [ 8.520706] systemd[1]: Initializing machine ID from random generator. [ 8.633601] random: ln: uninitialized urandom read (6 bytes read) [ 8.931910] random: systemd: uninitialized urandom read (16 bytes read) [ 8.936427] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 8.941473] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 8.951706] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.410409] device-mapper: uevent: version 1.0.3 [ 10.412981] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.652892] virtio_net virtio0 ens2: renamed from eth0 [ 11.773750] random: fast init done [ 12.116450] scsi host0: ata_piix [ 12.426680] scsi host1: ata_piix [ 12.455897] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.475576] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.532162] random: crng init done [ 17.533626] random: 7 urandom warning(s) missed due to ratelimiting [ 20.301643] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.280263] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.417995] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.002307] SELinux: Disabled at runtime. [ 26.088371] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.105959] systemd[1]: Detected virtualization kvm. [ 26.107944] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.394157] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.402805] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.414663] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.421597] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.425903] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.435930] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.444498] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ 28.558269] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 30.731821] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.222019] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.245694] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.065021] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.209949] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.530461] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 38.311716] NFS: Registering the id_resolver key type [ 38.314804] Key type id_resolver registered [ 38.317147] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg208-server login: [ 78.815758] libcfs: loading out-of-tree module taints kernel. [ 78.830580] Key type ._llcrypt registered [ 78.832077] Key type .llcrypt registered [ 78.875860] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 87.728677] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 88.363955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.371729] alg: No test for adler32 (adler32-zlib) [ 89.381802] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 89.748412] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 91.407135] Key type lgssc registered [ 92.241842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.936306] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 121.892892] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 127.856029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 127.864188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 128.990250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 129.008479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 129.051753] Lustre: lustre-MDT0000: new disk, initializing [ 129.118352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.131199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 130.938464] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 137.658185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 137.713312] Lustre: 6478:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 137.730205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 137.732635] Lustre: Skipped 1 previous similar message [ 137.775112] Lustre: lustre-MDT0001: new disk, initializing [ 137.801704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 137.819806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 137.825213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 139.448329] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 142.054337] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 146.769831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 146.890212] Lustre: lustre-OST0000: new disk, initializing [ 146.892989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 146.948065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 149.956482] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 152.592090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 152.598286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 152.644141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 157.637744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 157.690809] Lustre: lustre-OST0001: new disk, initializing [ 157.695124] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 157.727368] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 160.456686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 160.472636] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 160.518327] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 160.831974] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.844029] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 180.875241] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 184.522733] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing check_logdir /tmp/testlogs/ [ 187.801501] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing yml_node [ 190.123499] Lustre: DEBUG MARKER: Client: 2.17.52.127 [ 191.625479] Lustre: DEBUG MARKER: MDS: 2.17.52.127 [ 193.133602] Lustre: DEBUG MARKER: OSS: 2.17.52.127 [ 194.227363] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon May 4 11:55:10 EDT 2026 [ 204.700250] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 205.564738] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 219.615723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 219.618431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 219.623444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 222.175639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 222.176044] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 222.179217] Lustre: Skipped 2 previous similar messages [ 222.187153] Lustre: Skipped 2 previous similar messages [ 224.830896] Lustre: server umount lustre-MDT0000 complete [ 226.595910] LustreError: 6469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910144 with bad export cookie 9097588714756413705 [ 226.599128] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 226.610645] LustreError: 6469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 227.302844] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 227.303816] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 227.312265] Lustre: Skipped 1 previous similar message [ 227.317742] Lustre: Skipped 1 previous similar message [ 231.332166] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 231.335069] Lustre: Skipped 1 previous similar message [ 232.865195] Lustre: server umount lustre-MDT0001 complete [ 235.117357] Lustre: server umount lustre-OST0000 complete [ 237.220375] Lustre: server umount lustre-OST0001 complete [ 240.369834] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 244.889869] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 264.261141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 264.405068] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 264.419941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 264.461972] Lustre: lustre-MDT0000: new disk, initializing [ 264.506637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.518787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 266.790696] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 273.015493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 273.069310] Lustre: 14938:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 273.109684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 273.113331] Lustre: Skipped 1 previous similar message [ 273.180384] Lustre: lustre-MDT0001: new disk, initializing [ 273.251546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 273.271907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 273.278193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 275.308318] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 279.905330] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 281.759548] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 285.051312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 286.733176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 286.831098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 288.133917] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 292.544756] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 294.633981] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 294.736156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 296.466775] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 296.558169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 302.053183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 302.053623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.058176] LustreError: Skipped 1 previous similar message [ 302.068313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 302.075351] Lustre: Skipped 2 previous similar messages [ 303.489250] Lustre: server umount lustre-OST0000 complete [ 307.172146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.178655] Lustre: Skipped 2 previous similar messages [ 311.397290] Lustre: server umount lustre-MDT0000 complete [ 312.289104] LustreError: 14949:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.297757] LustreError: 14949:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 313.295403] LustreError: 16680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910230 with bad export cookie 9097588714756415238 [ 313.299332] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.300694] LustreError: 16680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 313.434520] Lustre: server umount lustre-MDT0001 complete [ 315.053925] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 11:57:11 (1777910231) [ 318.308294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 318.610687] LustreError: 18794:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.642455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 318.646348] Lustre: Skipped 1 previous similar message [ 320.804673] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 324.064511] LustreError: 18795:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.149264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 327.252376] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 330.172749] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 332.153888] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 335.644800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 338.658107] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 341.556782] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 355.300295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 355.302630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.306423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 355.315118] Lustre: Skipped 1 previous similar message [ 355.323385] Lustre: Skipped 3 previous similar messages [ 356.587447] Lustre: server umount lustre-OST0000 complete [ 364.847378] Lustre: server umount lustre-MDT0000 complete [ 365.536364] LustreError: 18789:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.542820] LustreError: 18789:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 366.787747] LustreError: 18774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910284 with bad export cookie 9097588714756416120 [ 366.788850] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 366.795496] LustreError: 18774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 366.970618] Lustre: server umount lustre-MDT0001 complete [ 372.637432] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 374.045516] Key type lgssc unregistered [ 374.198446] LNet: 21991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.205552] LNetError: 21991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 374.213672] LNet: Removed LNI 192.168.202.108@tcp [ 374.641138] Key type .llcrypt unregistered [ 374.642397] Key type ._llcrypt unregistered [ 388.969826] Key type ._llcrypt registered [ 388.971459] Key type .llcrypt registered [ 389.024880] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 389.475376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 389.493551] alg: No test for adler32 (adler32-zlib) [ 390.421249] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 390.566599] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 392.183162] Key type lgssc registered [ 392.837320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 397.702637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 397.716689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.000229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.100370] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 406.274916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.505355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 408.756255] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 412.028277] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 414.300804] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 418.611717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 418.837873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.133350] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 423.971137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 424.624236] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 439.264158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 439.267141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 439.271744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 441.315229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 441.315723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 441.316606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.317877] Lustre: Skipped 1 previous similar message [ 443.675774] Lustre: server umount lustre-OST0000 complete [ 446.433710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.435099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 446.438419] Lustre: Skipped 1 previous similar message [ 451.325225] Lustre: server umount lustre-MDT0000 complete [ 451.552149] LustreError: 23365:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.557033] LustreError: 23365:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 452.857340] LustreError: 23346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910370 with bad export cookie 8369101732353540641 [ 452.859949] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 452.861852] LustreError: 23346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 452.999803] Lustre: server umount lustre-MDT0001 complete [ 457.881794] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 459.160544] Key type lgssc unregistered [ 459.318700] LNet: 26980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.322539] LNetError: 26980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 459.330341] LNet: Removed LNI 192.168.202.108@tcp [ 459.683992] Key type .llcrypt unregistered [ 459.685659] Key type ._llcrypt unregistered [ 470.481126] Key type ._llcrypt registered [ 470.482362] Key type .llcrypt registered [ 470.533522] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 470.927755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 470.963445] alg: No test for adler32 (adler32-zlib) [ 471.844071] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 471.947791] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 473.543161] Key type lgssc registered [ 473.979852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 477.400438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 477.408796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.625446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 480.330679] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 483.971265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.157724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 485.770121] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 488.158719] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 489.693591] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 495.273850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 495.464723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.380051] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 501.153219] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 503.657117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 519.136678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 519.137243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.140280] LustreError: Skipped 1 previous similar message [ 519.142292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.145257] Lustre: Skipped 1 previous similar message [ 524.256986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 524.261467] Lustre: Skipped 2 previous similar messages [ 529.375111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 529.378666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.386338] Lustre: Skipped 1 previous similar message [ 529.450265] Lustre: server umount lustre-OST0000 complete [ 534.496969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.497508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 534.504251] Lustre: Skipped 1 previous similar message [ 537.385237] Lustre: server umount lustre-MDT0000 complete [ 539.100316] LustreError: 28332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910456 with bad export cookie 1027921182957437363 [ 539.104645] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.105042] LustreError: 28332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 539.265405] Lustre: server umount lustre-MDT0001 complete [ 543.387491] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 544.676086] Key type lgssc unregistered [ 544.816377] LNet: 31902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 544.820183] LNetError: 31902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 544.835905] LNet: Removed LNI 192.168.202.108@tcp [ 545.175803] Key type .llcrypt unregistered [ 545.177468] Key type ._llcrypt unregistered [ 554.936632] Key type ._llcrypt registered [ 554.937924] Key type .llcrypt registered [ 554.980924] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 555.423739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 555.433124] alg: No test for adler32 (adler32-zlib) [ 556.294949] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 556.377971] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 557.959152] Key type lgssc registered [ 558.364546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 561.532985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 561.540990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.719571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.279815] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 567.835799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.984704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 569.593813] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 571.754824] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 573.453864] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 576.411229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 576.580982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.216410] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 581.539367] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 587.881527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 598.499343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 598.503372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.505900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 598.508009] Lustre: Skipped 1 previous similar message [ 601.314607] Lustre: server umount lustre-OST0000 complete [ 603.617318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.620759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.623778] Lustre: Skipped 1 previous similar message [ 603.627946] Lustre: Skipped 1 previous similar message [ 608.736067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 608.740118] Lustre: Skipped 2 previous similar messages [ 609.195205] Lustre: server umount lustre-MDT0000 complete [ 610.925706] LustreError: 33257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910528 with bad export cookie 18200073094001992453 [ 610.926125] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.930533] LustreError: 33257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 611.062555] Lustre: server umount lustre-MDT0001 complete [ 615.301983] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 616.708822] Key type lgssc unregistered [ 616.858136] LNet: 36465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.861920] LNetError: 36465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 616.870422] LNet: Removed LNI 192.168.202.108@tcp [ 617.249190] Key type .llcrypt unregistered [ 617.250956] Key type ._llcrypt unregistered [ 628.658626] Key type ._llcrypt registered [ 628.659701] Key type .llcrypt registered [ 628.719951] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 629.146907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 629.200155] alg: No test for adler32 (adler32-zlib) [ 630.101715] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 630.221086] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 631.823229] Key type lgssc registered [ 632.440787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 636.394113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 636.404511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.721254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 639.886850] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 644.773841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.967776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 646.808500] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 649.537953] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 651.206322] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 654.563221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 654.734890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 657.378467] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 659.927400] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 662.950370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 671.199799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 671.202682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 671.207717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 671.820919] Lustre: lustre-OST0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 673.249628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 673.250592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 673.250817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.259403] Lustre: Skipped 1 previous similar message [ 675.172612] Lustre: server umount lustre-OST0000 complete [ 691.170804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 691.174145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.180606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 694.242018] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.000451] Lustre: server umount lustre-MDT0000 complete [ 696.572314] LustreError: 38556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910614 with bad export cookie 15271635304385556121 [ 696.575641] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 696.577137] LustreError: 38556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 702.773963] Lustre: server umount lustre-MDT0001 complete [ 707.016335] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 708.299338] Key type lgssc unregistered [ 708.471835] LNet: 41190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.475314] LNetError: 41190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 708.487513] LNet: Removed LNI 192.168.202.108@tcp [ 708.854523] Key type .llcrypt unregistered [ 708.856265] Key type ._llcrypt unregistered [ 719.904627] Key type ._llcrypt registered [ 719.906082] Key type .llcrypt registered [ 719.952096] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 720.432819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 720.459468] alg: No test for adler32 (adler32-zlib) [ 721.392564] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 721.501564] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 723.111220] Key type lgssc registered [ 723.566731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 726.856627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 726.865653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.063060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.530207] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 732.774462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 732.902902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 734.282736] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 736.353589] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 737.921868] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 741.000231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 741.129717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 743.400128] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 745.632193] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 753.447186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 761.425622] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 763.872593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 763.874174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 763.875702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 763.888262] Lustre: Skipped 2 previous similar messages [ 765.363239] Lustre: server umount lustre-MDT0000 complete [ 766.532691] LustreError: 44709:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 766.540284] LustreError: 44709:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 767.317367] LustreError: 42544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777910684 with bad export cookie 8565651085783391294 [ 767.322719] LustreError: 42544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 767.322734] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.992385] LustreError: 43273:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 768.992743] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.003801] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 769.005950] Lustre: Skipped 2 previous similar messages [ 771.651112] LustreError: 42558:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 771.662356] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.8@tcp (stopping) [ 773.515745] Lustre: server umount lustre-MDT0001 complete [ 800.223121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 800.288321] Lustre: server umount lustre-OST0000 complete [ 808.143682] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 809.556328] Key type lgssc unregistered [ 809.731407] LNet: 46085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.734818] LNetError: 46085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.744688] LNet: Removed LNI 192.168.202.108@tcp [ 810.131436] Key type .llcrypt unregistered [ 810.133050] Key type ._llcrypt unregistered [ 821.546522] Key type ._llcrypt registered [ 821.548311] Key type .llcrypt registered [ 821.600416] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 822.086861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.116731] alg: No test for adler32 (adler32-zlib) [ 823.034059] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 823.151870] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 824.751226] Key type lgssc registered [ 825.214498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 828.341842] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 828.350599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 843.807314] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 848.927149] LustreError: 47438:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 875.615206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.756630] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 879.800223] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1042.488203] Lustre: server umount lustre-OST0000 complete [ 1048.430942] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1049.586186] Key type lgssc unregistered [ 1049.724456] LNet: 48982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.727462] LNetError: 48982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.740506] LNet: Removed LNI 192.168.202.108@tcp [ 1050.061061] Key type .llcrypt unregistered [ 1050.062512] Key type ._llcrypt unregistered [ 1059.866925] Key type ._llcrypt registered [ 1059.869321] Key type .llcrypt registered [ 1059.921280] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1060.483309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1060.494277] alg: No test for adler32 (adler32-zlib) [ 1061.368551] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1061.470634] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1063.063176] Key type lgssc registered [ 1063.505578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1066.659790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1066.668355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.893621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.692408] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1073.309965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.536346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1075.449588] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1077.989882] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1079.547785] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1082.874182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1083.122224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1086.119959] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1088.963461] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1090.043925] Lustre: server umount lustre-OST0000 complete [ 1093.602336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.606039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.611302] Lustre: Skipped 1 previous similar message [ 1097.696591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.699420] Lustre: Skipped 2 previous similar messages [ 1097.824979] Lustre: server umount lustre-MDT0000 complete [ 1099.692758] LustreError: 51074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911017 with bad export cookie 7089511353115403492 [ 1099.699474] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1099.864130] Lustre: server umount lustre-MDT0001 complete [ 1104.557327] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1106.014645] Key type lgssc unregistered [ 1106.167656] LNet: 53524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.170273] LNetError: 53524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.181323] LNet: Removed LNI 192.168.202.108@tcp [ 1106.569355] Key type .llcrypt unregistered [ 1106.570996] Key type ._llcrypt unregistered [ 1117.296255] Key type ._llcrypt registered [ 1117.297366] Key type .llcrypt registered [ 1117.353701] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1117.708325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1117.725063] alg: No test for adler32 (adler32-zlib) [ 1118.621587] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1118.722135] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1120.311221] Key type lgssc registered [ 1120.742170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1124.154608] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1124.166591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1139.615447] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1144.735299] LustreError: 54871:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 1171.359126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.669166] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1175.811704] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1178.535936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.737332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.763231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1180.281760] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1183.974897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.149287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1185.784925] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1188.036599] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1189.556714] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1191.397281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.402942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1191.405686] Lustre: Skipped 1 previous similar message [ 1196.510606] Lustre: server umount lustre-OST0000 complete [ 1214.943872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1214.950042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.954783] Lustre: Skipped 1 previous similar message [ 1214.957119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.968066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.968536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.185865] Lustre: server umount lustre-MDT0000 complete [ 1218.716566] LustreError: 54878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911136 with bad export cookie 8981785492374447443 [ 1218.718047] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1218.721775] LustreError: 54878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1232.863123] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1232.950400] Lustre: server umount lustre-MDT0001 complete [ 1236.642356] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1237.727372] Key type lgssc unregistered [ 1237.855480] LNet: 58253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.858750] LNetError: 58253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.866318] LNet: Removed LNI 192.168.202.108@tcp [ 1238.166639] Key type .llcrypt unregistered [ 1238.167952] Key type ._llcrypt unregistered [ 1247.340419] Key type ._llcrypt registered [ 1247.342171] Key type .llcrypt registered [ 1247.389085] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1247.808971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1247.857655] alg: No test for adler32 (adler32-zlib) [ 1248.741920] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1248.837327] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1250.431282] Key type lgssc registered [ 1250.948934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1253.746525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1253.754688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1254.930852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1256.256554] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1259.155372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.280889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1260.553308] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1262.380315] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1263.631270] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1266.037702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1266.169457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.136145] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1269.869312] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1280.550635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1289.696331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1289.699849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.705723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1290.722057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1290.722764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1290.723024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.730324] Lustre: Skipped 1 previous similar message [ 1292.489730] Lustre: server umount lustre-OST0000 complete [ 1295.328188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1295.330374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.335188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1299.801216] Lustre: server umount lustre-MDT0000 complete [ 1300.959877] LustreError: 61775:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.966127] LustreError: 61775:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1301.058354] LustreError: 59605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911218 with bad export cookie 4148628314085053491 [ 1301.061470] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1301.062010] LustreError: 59605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1301.182421] Lustre: server umount lustre-MDT0001 complete [ 1304.823812] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1305.871638] Key type lgssc unregistered [ 1305.999480] LNet: 62821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.002428] LNetError: 62821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.010294] LNet: Removed LNI 192.168.202.108@tcp [ 1306.310563] Key type .llcrypt unregistered [ 1306.311574] Key type ._llcrypt unregistered [ 1316.116843] Key type ._llcrypt registered [ 1316.118297] Key type .llcrypt registered [ 1316.162684] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1316.541339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1316.563551] alg: No test for adler32 (adler32-zlib) [ 1317.436358] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1317.527655] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1319.111223] Key type lgssc registered [ 1319.526950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1322.487153] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 12:13:59 (1777911239) [ 1325.044311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1325.050929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1326.254236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1327.831373] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1331.359618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.529572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1333.188391] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1335.246109] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1336.668413] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1339.149667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1339.289266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.287739] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1343.239238] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1350.566705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1355.999984] Lustre: Failing over lustre-MDT0000 [ 1356.216330] Lustre: server umount lustre-MDT0000 complete [ 1357.280752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1357.283704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.289100] LustreError: 66528:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.295402] LustreError: 66528:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1358.929493] LustreError: 66634:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.756626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.803267] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1359.861699] Lustre: *** cfs_fail_loc=135, val=0*** [ 1359.903101] LustreError: 67256:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1359.906476] LustreError: 67256:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.108@tcp: cfg command failed: rc = -2 [ 1359.909833] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1359.909833] [ 1359.912779] LustreError: MGC192.168.202.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1359.918752] LustreError: 67243:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1359.921610] LustreError: 67243:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1359.925616] LustreError: 67243:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1359.929084] LustreError: 67243:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1359.938873] Lustre: server umount lustre-MDT0000 complete [ 1359.940192] LustreError: 67243:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1360.865587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.865990] LustreError: 64485:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.869218] Lustre: Skipped 1 previous similar message [ 1360.874646] LustreError: 64485:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1362.341953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.396175] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1362.501305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1362.521412] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1363.877642] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1364.034152] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1367.525678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1367.534495] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1367.553637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1372.127902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1372.130386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1372.135901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1372.644781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1374.611396] Lustre: server umount lustre-OST0000 complete [ 1377.761267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1377.765512] Lustre: Skipped 2 previous similar messages [ 1377.767402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1377.767653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1377.770107] LustreError: Skipped 1 previous similar message [ 1377.772629] Lustre: Skipped 1 previous similar message [ 1382.165051] Lustre: server umount lustre-MDT0000 complete [ 1382.880055] LustreError: 64485:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.885970] LustreError: 64485:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1383.458344] LustreError: 64464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911300 with bad export cookie 3630555469932553580 [ 1383.461611] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1383.462217] LustreError: 64464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1383.582955] Lustre: server umount lustre-MDT0001 complete [ 1387.174619] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1388.321271] Key type lgssc unregistered [ 1388.446459] LNet: 68995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.448751] LNetError: 68995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.458348] LNet: Removed LNI 192.168.202.108@tcp [ 1388.780430] Key type .llcrypt unregistered [ 1388.781832] Key type ._llcrypt unregistered [ 1397.714171] Key type ._llcrypt registered [ 1397.715450] Key type .llcrypt registered [ 1397.762161] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1398.118536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1398.151800] alg: No test for adler32 (adler32-zlib) [ 1399.029879] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1399.114060] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1400.703243] Key type lgssc registered [ 1401.155107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1403.987670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1403.995046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.141401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.422445] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1409.326917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1409.450448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1410.797844] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1412.724841] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1414.076393] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1416.394299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1416.511950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.408268] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1420.323132] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1429.861086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1432.800581] Lustre: Failing over lustre-MDT0000 [ 1433.029399] Lustre: server umount lustre-MDT0000 complete [ 1435.106505] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.111191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1435.114088] LustreError: 70367:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.121080] LustreError: 70367:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 1435.720745] LustreError: 70361:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.313704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.360634] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1436.421533] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1436.483163] LustreError: 73132:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1436.489064] LustreError: 73132:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.108@tcp: cfg command failed: rc = -9 [ 1436.491864] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1436.491864] [ 1436.495434] LustreError: MGC192.168.202.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1436.500725] LustreError: 73121:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1436.503522] LustreError: 73121:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 1436.506879] LustreError: 73121:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1436.509814] LustreError: 73121:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1436.522896] Lustre: server umount lustre-MDT0000 complete [ 1436.524280] LustreError: 73121:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1438.834479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.880797] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1438.978207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.006382] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1440.361808] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1440.834240] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1444.327143] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1444.337646] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1444.356208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1449.442145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.443069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1449.447798] Lustre: Skipped 2 previous similar messages [ 1449.454491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.456611] Lustre: Skipped 1 previous similar message [ 1451.478867] Lustre: server umount lustre-OST0000 complete [ 1454.560662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1454.561076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1454.561631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1454.564140] LustreError: Skipped 1 previous similar message [ 1454.567360] Lustre: Skipped 2 previous similar messages [ 1459.680312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1459.682739] Lustre: Skipped 2 previous similar messages [ 1464.800086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1464.802123] Lustre: Skipped 1 previous similar message [ 1467.359137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1467.424363] Lustre: server umount lustre-MDT0000 complete [ 1468.721161] LustreError: 70348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911386 with bad export cookie 8961258638339478073 [ 1468.723907] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1468.725967] LustreError: 70348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1468.831676] Lustre: server umount lustre-MDT0001 complete [ 1472.376606] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1473.381595] Key type lgssc unregistered [ 1473.510465] LNet: 74872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.512854] LNetError: 74872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1473.525355] LNet: Removed LNI 192.168.202.108@tcp [ 1473.831851] Key type .llcrypt unregistered [ 1473.833566] Key type ._llcrypt unregistered [ 1477.873355] Key type ._llcrypt registered [ 1477.874652] Key type .llcrypt registered [ 1477.911959] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1478.010424] Key type .llcrypt unregistered [ 1478.011351] Key type ._llcrypt unregistered [ 1486.788713] Key type ._llcrypt registered [ 1486.789943] Key type .llcrypt registered [ 1486.824395] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1487.116223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1487.151059] alg: No test for adler32 (adler32-zlib) [ 1488.011691] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1488.095663] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1489.679221] Key type lgssc registered [ 1490.031312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1492.634155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1492.642347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.796891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1495.211382] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1498.051210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.174430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1499.615233] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1501.534061] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1502.814539] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1505.323154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1505.488326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.493578] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1509.229411] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1518.818960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1523.167902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1523.170049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.173568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1524.192610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1524.193770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.195055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1528.465377] Lustre: server umount lustre-OST0000 complete [ 1533.919954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1533.922629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.926738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1533.928686] Lustre: Skipped 1 previous similar message [ 1535.835175] Lustre: server umount lustre-MDT0000 complete [ 1537.100580] LustreError: 76740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911454 with bad export cookie 14626690262673532507 [ 1537.103392] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1537.104818] LustreError: 76740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1537.231617] Lustre: server umount lustre-MDT0001 complete [ 1540.732554] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1541.746948] Key type lgssc unregistered [ 1541.865343] LNet: 79912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.868269] LNetError: 79912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.877312] LNet: Removed LNI 192.168.202.108@tcp [ 1542.155253] Key type .llcrypt unregistered [ 1542.156574] Key type ._llcrypt unregistered [ 1550.838541] Key type ._llcrypt registered [ 1550.839751] Key type .llcrypt registered [ 1550.876838] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1551.187550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1551.196959] alg: No test for adler32 (adler32-zlib) [ 1552.064359] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1552.152778] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1553.727131] Key type lgssc registered [ 1554.116168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1556.810702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1556.817089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1557.964685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.355387] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1562.335688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.464349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1563.740609] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1565.568805] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1566.747907] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1568.945831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1569.061379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1570.919435] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1572.638638] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1583.396412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1592.800179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1592.802878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.807146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1593.824603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1593.825139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1593.825382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1593.835487] Lustre: Skipped 1 previous similar message [ 1596.694237] Lustre: server umount lustre-OST0000 complete [ 1598.431953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1598.435435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.441595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1604.064033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1604.066815] Lustre: Skipped 2 previous similar messages [ 1609.183908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.186500] Lustre: Skipped 1 previous similar message [ 1612.255360] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1612.315465] Lustre: server umount lustre-MDT0000 complete [ 1613.641428] LustreError: 81263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911531 with bad export cookie 11657909797442263156 [ 1613.643768] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1613.645163] LustreError: 81263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1613.765300] Lustre: server umount lustre-MDT0001 complete [ 1617.225259] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1618.221228] Key type lgssc unregistered [ 1618.337310] LNet: 84481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1618.339599] LNetError: 84481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1618.345248] LNet: Removed LNI 192.168.202.108@tcp [ 1618.638251] Key type .llcrypt unregistered [ 1618.639698] Key type ._llcrypt unregistered [ 1627.116807] Key type ._llcrypt registered [ 1627.118008] Key type .llcrypt registered [ 1627.156452] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1627.551892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1627.568941] alg: No test for adler32 (adler32-zlib) [ 1628.415112] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1628.496493] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1630.079152] Key type lgssc registered [ 1630.443367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1632.923641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1632.929422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.086987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1635.424543] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1638.495531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.621846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1640.026592] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1641.916722] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1643.169259] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1645.580406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1645.718394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1647.693615] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1649.637985] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1659.047202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1664.481034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1664.482263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.483984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1664.486355] Lustre: Skipped 1 previous similar message [ 1669.131448] Lustre: server umount lustre-OST0000 complete [ 1674.720745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1674.720843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.721475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.721480] Lustre: Skipped 1 previous similar message [ 1674.723245] LustreError: Skipped 1 previous similar message [ 1674.726527] Lustre: Skipped 1 previous similar message [ 1676.563176] Lustre: server umount lustre-MDT0000 complete [ 1677.805159] LustreError: 85835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911595 with bad export cookie 15886899963208559757 [ 1677.807814] LustreError: 85835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1677.808318] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1677.929256] Lustre: server umount lustre-MDT0001 complete [ 1681.319565] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1682.334411] Key type lgssc unregistered [ 1682.464444] LNet: 89052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.467232] LNetError: 89052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.475410] LNet: Removed LNI 192.168.202.108@tcp [ 1682.760301] Key type .llcrypt unregistered [ 1682.761581] Key type ._llcrypt unregistered [ 1691.271935] Key type ._llcrypt registered [ 1691.273475] Key type .llcrypt registered [ 1691.309370] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1691.612341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1691.646790] alg: No test for adler32 (adler32-zlib) [ 1692.501037] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1692.579144] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1694.159224] Key type lgssc registered [ 1694.506404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1696.991884] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1696.998430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1712.415299] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1717.535280] LustreError: 90398:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 1744.159250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.241473] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1748.055267] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1749.958938] Lustre: server umount lustre-OST0000 complete [ 1752.679457] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 12:21:09 (1777911669) [ 1754.869217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1755.015535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1756.225765] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1758.972323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1759.116927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1760.515509] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1762.332360] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1763.490953] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1765.687959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1765.808792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1767.678466] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1769.463134] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1780.135677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1789.407907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1789.412051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.418507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1790.433271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1790.435108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1790.435229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.322907] Lustre: server umount lustre-OST0000 complete [ 1800.160033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1800.162327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.167032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.168629] Lustre: Skipped 1 previous similar message [ 1801.753943] Lustre: server umount lustre-MDT0000 complete [ 1802.992227] LustreError: 94250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911720 with bad export cookie 4221640759354777207 [ 1802.995266] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1802.998557] LustreError: 94250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1803.112625] Lustre: server umount lustre-MDT0001 complete [ 1806.672918] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1807.765984] Key type lgssc unregistered [ 1807.900393] LNet: 95273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.904503] LNetError: 95273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.913337] LNet: Removed LNI 192.168.202.108@tcp [ 1808.218133] Key type .llcrypt unregistered [ 1808.219489] Key type ._llcrypt unregistered [ 1821.263788] Key type ._llcrypt registered [ 1821.264938] Key type .llcrypt registered [ 1821.296521] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1821.657161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1821.668245] alg: No test for adler32 (adler32-zlib) [ 1822.512278] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1822.584991] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1824.159162] Key type lgssc registered [ 1824.487137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1826.840307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1826.846286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.992222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.166413] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1831.755654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.859144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1833.074982] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1834.677247] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1835.822176] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1837.857673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1837.965478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1839.014221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1839.660372] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1841.251339] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1847.266523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1847.266819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.267718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1847.275953] Lustre: Skipped 1 previous similar message [ 1850.696230] Lustre: server umount lustre-OST0000 complete [ 1852.384516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1852.384884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1852.391702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1852.393201] Lustre: Skipped 2 previous similar messages [ 1857.504267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1857.505898] Lustre: Skipped 1 previous similar message [ 1862.624087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1862.626155] Lustre: Skipped 1 previous similar message [ 1866.207103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1866.262759] Lustre: server umount lustre-MDT0000 complete [ 1866.720927] LustreError: 97415:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1866.726304] LustreError: 97415:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1867.446983] LustreError: 97397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911784 with bad export cookie 15729907870333645648 [ 1867.449178] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1867.451196] LustreError: 97397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1867.564588] Lustre: server umount lustre-MDT0001 complete [ 1870.801526] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1871.794567] Key type lgssc unregistered [ 1871.915375] LNet: 100588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.917759] LNetError: 100588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.926274] LNet: Removed LNI 192.168.202.108@tcp [ 1872.201416] Key type .llcrypt unregistered [ 1872.202371] Key type ._llcrypt unregistered [ 1878.998481] Key type ._llcrypt registered [ 1878.999563] Key type .llcrypt registered [ 1879.032292] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1879.338443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1879.381648] alg: No test for adler32 (adler32-zlib) [ 1880.231763] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1880.317202] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1881.895147] Key type lgssc registered [ 1882.217625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1884.611867] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1884.618657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1900.063275] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1905.183184] LustreError: 101790:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 1931.807204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.550560] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1935.201484] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1937.201615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.288838] LustreError: MGC192.168.202.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1937.293210] LustreError: 102700:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1937.295689] LustreError: 102700:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1937.299092] LustreError: 102700:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1937.323038] Lustre: server umount lustre-MDT0000 complete [ 1937.324204] LustreError: 102700:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1950.209399] Lustre: server umount lustre-OST0000 complete [ 1953.001327] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 1955.775319] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1966.291444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.372778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1966.384910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1966.423291] Lustre: lustre-MDT0000: new disk, initializing [ 1966.450859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1966.457355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1967.625949] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1971.539799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.576514] Lustre: 105452:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1971.596800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1971.598168] Lustre: Skipped 1 previous similar message [ 1971.628757] Lustre: lustre-MDT0001: new disk, initializing [ 1971.657488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1971.668656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1971.672082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1972.837205] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1975.736573] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1976.811941] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1978.905230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1979.009194] Lustre: lustre-OST0000: new disk, initializing [ 1979.011182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1979.047182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.751758] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1981.945603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1981.949148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1981.982648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1983.617408] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1984.809564] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1984.868954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1986.074685] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1986.148486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1987.042135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1987.042753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.045944] LustreError: Skipped 1 previous similar message [ 1987.049924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.053733] Lustre: Skipped 1 previous similar message [ 1992.163038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1992.165639] Lustre: Skipped 1 previous similar message [ 1997.281299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1997.283391] Lustre: Skipped 1 previous similar message [ 2000.863138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2000.919636] Lustre: server umount lustre-OST0000 complete [ 2002.400379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2002.400534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.401120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.409067] Lustre: Skipped 2 previous similar messages [ 2007.519863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.522696] Lustre: Skipped 2 previous similar messages [ 2008.273895] Lustre: server umount lustre-MDT0000 complete [ 2009.395381] LustreError: 105444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911926 with bad export cookie 16157714445179944394 [ 2009.398630] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2009.401100] LustreError: 105444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2009.512734] Lustre: server umount lustre-MDT0001 complete [ 2012.266480] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 12:25:28 (1777911928) [ 2013.364367] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2018.494730] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 2021.535329] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2033.049206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2033.136160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2033.145543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2033.179335] Lustre: lustre-MDT0000: new disk, initializing [ 2033.203258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2033.209638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2034.462691] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2038.511495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.555961] Lustre: 111900:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2038.580351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2038.582850] Lustre: Skipped 1 previous similar message [ 2038.621018] Lustre: lustre-MDT0001: new disk, initializing [ 2038.663828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2038.667326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2039.968414] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2042.999520] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2044.142754] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2046.368468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2046.492186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.494328] Lustre: Skipped 1 previous similar message [ 2048.356755] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2049.588039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2049.591040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2049.622188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2051.372427] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2052.658107] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2052.730466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2054.062530] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2054.148518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2059.745918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.746126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2059.752177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2059.753921] LustreError: Skipped 1 previous similar message [ 2059.758747] Lustre: Skipped 1 previous similar message [ 2060.940148] Lustre: server umount lustre-OST0000 complete [ 2064.351891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2064.355647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.360044] Lustre: Skipped 1 previous similar message [ 2068.256093] Lustre: server umount lustre-MDT0000 complete [ 2069.589133] LustreError: 111892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777911987 with bad export cookie 16157714445179945262 [ 2069.592495] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2069.594808] LustreError: 111892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2069.703810] Lustre: server umount lustre-MDT0001 complete [ 2072.503389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.676583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.977718] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2076.751734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.195665] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2079.883891] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2081.080593] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2083.255271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2083.296497] LustreError: 117295:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.301661] LustreError: 117295:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2085.159244] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2086.898073] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2095.419421] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2097.636678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2097.638540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.640130] LustreError: Skipped 1 previous similar message [ 2097.643185] Lustre: Skipped 1 previous similar message [ 2097.643674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2097.647917] Lustre: Skipped 3 previous similar messages [ 2102.543986] Lustre: server umount lustre-OST0000 complete [ 2107.872661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.873349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2107.876442] Lustre: Skipped 1 previous similar message [ 2118.111214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2118.172272] Lustre: server umount lustre-MDT0000 complete [ 2119.452121] LustreError: 116422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912036 with bad export cookie 16157714445179946130 [ 2119.454873] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2119.456470] LustreError: 116422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2119.577091] Lustre: server umount lustre-MDT0001 complete [ 2122.627432] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2123.553659] Key type lgssc unregistered [ 2123.676297] LNet: 118974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.678330] LNetError: 118974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.685298] LNet: Removed LNI 192.168.202.108@tcp [ 2123.961993] Key type .llcrypt unregistered [ 2123.962994] Key type ._llcrypt unregistered [ 2129.241526] Key type ._llcrypt registered [ 2129.243091] Key type .llcrypt registered [ 2129.277275] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 2135.271916] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2135.717541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2135.733881] alg: No test for adler32 (adler32-zlib) [ 2136.580724] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 2136.658056] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2138.239155] Key type lgssc registered [ 2138.629919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2149.830955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2149.836114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2150.916915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2150.928488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2150.962387] Lustre: lustre-MDT0000: new disk, initializing [ 2150.986115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.991309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2152.203649] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2156.082746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.119951] Lustre: 122064:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2156.141548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2156.143062] Lustre: Skipped 1 previous similar message [ 2156.173528] Lustre: lustre-MDT0001: new disk, initializing [ 2156.202435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2156.211783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2156.214652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2157.366663] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2160.231604] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2161.314941] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2163.367572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.454091] Lustre: lustre-OST0000: new disk, initializing [ 2163.455425] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2163.477506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.069612] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2167.991593] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2168.311519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2168.314932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2168.347158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2169.277073] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2169.345736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2170.651098] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2170.725787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2173.409358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2173.409380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.412811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2173.417068] Lustre: Skipped 1 previous similar message [ 2177.488088] Lustre: server umount lustre-OST0000 complete [ 2182.111911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2182.114036] LustreError: Skipped 1 previous similar message [ 2182.115493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.119401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.120853] Lustre: Skipped 1 previous similar message [ 2183.648152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2183.648400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.796338] Lustre: server umount lustre-MDT0000 complete [ 2186.090091] LustreError: 123806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912103 with bad export cookie 2262816677276196369 [ 2186.093536] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2186.094408] LustreError: 123806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2186.219840] Lustre: server umount lustre-MDT0001 complete [ 2188.973632] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 12:28:25 (1777912105) [ 2191.162162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.313560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.491572] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2195.072077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.415842] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2198.085331] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2199.226721] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2199.875333] LustreError: 126104:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2199.880151] LustreError: 126104:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2199.884595] LustreError: 126104:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2200.544496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.544543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2200.545060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2200.545066] Lustre: Skipped 1 previous similar message [ 2200.548154] Lustre: Skipped 1 previous similar message [ 2204.640349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2204.642601] Lustre: Skipped 2 previous similar messages [ 2214.367218] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2214.427390] Lustre: server umount lustre-MDT0000 complete [ 2214.879899] LustreError: 126074:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.885259] LustreError: 126074:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2215.650059] LustreError: 126055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912133 with bad export cookie 2262816677276197237 [ 2215.652779] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2215.654433] LustreError: 126055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2215.764996] Lustre: server umount lustre-MDT0001 complete [ 2218.360750] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 12:28:54 (1777912134) [ 2220.382532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2235.807486] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2240.927496] LustreError: 128260:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 2266.527493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.531300] Lustre: Skipped 1 previous similar message [ 2268.391422] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2270.239149] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2270.941990] Lustre: server umount lustre-OST0000 complete [ 2273.327190] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 12:29:49 (1777912189) [ 2275.278011] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.403358] LustreError: 129591:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.421864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.507115] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2279.263921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.509151] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2282.140915] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2283.213980] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2285.229066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2286.959420] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2288.546870] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2294.753515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.754605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2294.754639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2294.754643] Lustre: Skipped 2 previous similar messages [ 2294.757123] Lustre: Skipped 2 previous similar messages [ 2296.975919] Lustre: server umount lustre-MDT0000 complete [ 2298.057811] LustreError: 129616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912215 with bad export cookie 2262816677276197867 [ 2298.060724] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2298.061311] LustreError: 129616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2298.153226] Lustre: server umount lustre-MDT0001 complete [ 2309.452745] Lustre: server umount lustre-OST0000 complete [ 2311.775120] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 12:30:28 (1777912228) [ 2313.762328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.902378] LustreError: 132898:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.921271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2313.924190] Lustre: Skipped 2 previous similar messages [ 2315.013308] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2317.552642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.933698] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2320.711163] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2321.973276] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2324.164820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2326.191210] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2328.003865] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2329.327303] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2338.597114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2339.593542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2340.942520] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2341.016549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2343.905937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2343.906045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.908375] LustreError: Skipped 1 previous similar message [ 2343.913062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2343.916359] Lustre: Skipped 3 previous similar messages [ 2347.800194] Lustre: server umount lustre-OST0000 complete [ 2363.359191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2363.427485] Lustre: server umount lustre-MDT0000 complete [ 2364.383988] LustreError: 132899:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.387792] LustreError: 132899:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2364.570314] LustreError: 132878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912282 with bad export cookie 2262816677276198924 [ 2364.573293] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2364.574564] LustreError: 132878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2364.680102] Lustre: server umount lustre-MDT0001 complete [ 2367.178510] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 12:31:23 (1777912283) [ 2369.338438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2384.799458] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2389.919425] LustreError: 136749:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 2415.519313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.521854] Lustre: Skipped 2 previous similar messages [ 2417.200154] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2418.762196] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2420.761122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2420.880495] LustreError: 136774:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.921632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2421.987120] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2424.518021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.725285] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2427.274440] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2428.323224] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2429.468187] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2429.530503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2430.624899] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2430.686507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2435.041875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2435.043076] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.045517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2435.048825] Lustre: Skipped 5 previous similar messages [ 2435.052430] Lustre: Skipped 8 previous similar messages [ 2437.394249] Lustre: server umount lustre-MDT0000 complete [ 2438.478025] LustreError: 136756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912355 with bad export cookie 2262816677276199932 [ 2438.480984] LustreError: 136756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2438.481264] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2438.584225] Lustre: server umount lustre-MDT0001 complete [ 2449.937289] Lustre: server umount lustre-OST0000 complete [ 2452.698124] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 12:32:49 (1777912369) [ 2454.887924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2470.303545] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2475.423479] LustreError: 140565:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 2501.023386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.026031] Lustre: Skipped 2 previous similar messages [ 2502.853382] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2504.653257] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2506.865608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.997892] LustreError: 140590:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.036728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2508.144341] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2510.664367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2511.919202] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2513.517324] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2514.540888] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2516.617575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2516.672512] Lustre: lustre-OST0001: new disk, initializing [ 2516.674257] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2518.528116] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2519.802508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2519.805625] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2519.835392] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2521.670197] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2523.020277] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2523.085638] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2524.417206] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2524.491286] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2525.154685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.157101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2525.158573] Lustre: Skipped 1 previous similar message [ 2531.195427] Lustre: server umount lustre-OST0000 complete [ 2535.392942] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2535.394815] LustreError: Skipped 1 previous similar message [ 2535.394951] LustreError: 140582:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.399685] LustreError: 140582:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2538.582285] Lustre: server umount lustre-OST0001 complete [ 2554.335100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2554.400485] Lustre: server umount lustre-MDT0000 complete [ 2555.567520] LustreError: 143092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912473 with bad export cookie 2262816677276200884 [ 2555.570190] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2555.571471] LustreError: 143092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2555.681578] Lustre: server umount lustre-MDT0001 complete [ 2562.907102] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 12:34:39 (1777912479) [ 2563.414206] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2563.883736] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2564.393859] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 12:34:40 (1777912480) [ 2566.427415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.514919] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2566.524761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2567.781628] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2570.414992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2570.442535] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2570.452719] Lustre: 146500:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2571.731238] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2573.421543] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2574.600732] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2576.760788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2576.829331] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2578.662067] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2580.328489] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2580.684689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2581.577906] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2581.640733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2582.821468] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2582.893056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2586.081064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2586.083265] LustreError: Skipped 1 previous similar message [ 2589.579279] Lustre: server umount lustre-OST0000 complete [ 2603.221637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2605.092678] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2606.436617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.440028] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2606.440041] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.749531] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2608.224441] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2608.295826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2609.556957] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2609.626895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2610.849274] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e4e468af000.ost_server_uuid 50 [ 2611.364371] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e4e468af000.ost_server_uuid in FULL state after 0 sec [ 2611.418833] LustreError: 147633:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.8@tcp) returned error from blocking AST (req@000000001415dd24 x1864275910112768 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b2d03f05/0x1f6723c770e0b900 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.8@tcp remote: 0xc19982451bf9a24b expref: 5 pid: 147633 timeout: 2711 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2611.428806] LustreError: lustre-MDT0000: A client on nid 192.168.202.8@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2611.431265] LustreError: 146495:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.8@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b2d03f05/0x1f6723c770e0b900 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.8@tcp remote: 0xc19982451bf9a24b expref: 6 pid: 147633 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2616.802234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.805680] Lustre: Skipped 7 previous similar messages [ 2618.508760] Lustre: server umount lustre-OST0000 complete [ 2634.207172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2634.268901] Lustre: server umount lustre-MDT0000 complete [ 2635.474985] LustreError: 149397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912552 with bad export cookie 2262816677276202067 [ 2635.478176] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2635.478391] LustreError: 149397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2641.639674] Lustre: server umount lustre-MDT0001 complete [ 2644.970306] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2645.952745] Key type lgssc unregistered [ 2646.089394] LNet: 151970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2646.091992] LNetError: 151970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2646.101335] LNet: Removed LNI 192.168.202.108@tcp [ 2646.391556] Key type .llcrypt unregistered [ 2646.392647] Key type ._llcrypt unregistered [ 2654.681144] Key type ._llcrypt registered [ 2654.682262] Key type .llcrypt registered [ 2654.717695] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2655.054227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2655.076064] alg: No test for adler32 (adler32-zlib) [ 2655.918650] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 2655.996841] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2657.575103] Key type lgssc registered [ 2657.921379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2660.392110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2660.398298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.554474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.813873] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2665.462436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.584546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2666.784237] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2668.404728] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2669.598177] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2671.721248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2671.832913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2672.871299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2673.625223] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2675.261958] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2678.327460] Lustre: Failing over lustre-MDT0000 [ 2678.462021] Lustre: server umount lustre-MDT0000 complete [ 2680.799755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2680.801794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.805434] LustreError: 154066:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.809792] LustreError: 154066:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2681.312308] LustreError: 153354:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.313489] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.318048] LustreError: 153354:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2681.321779] Lustre: Skipped 1 previous similar message [ 2686.433633] LustreError: 153355:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.437551] LustreError: 153355:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2689.288330] LustreError: 153336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.8@tcp arrived at 1777912606 with bad export cookie 18416820932562160310 [ 2691.552882] LustreError: 154066:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.562335] LustreError: 154066:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2691.702390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.746967] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2691.842938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.876888] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2693.201800] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2693.937607] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2693.941517] Lustre: lustre-MDT0000: Denying connection for new client d1e8af12-6d83-49d3-b81a-2e0e16e1475d (at 192.168.202.8@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2697.194690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2701.791858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2701.795092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2701.800617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2702.304686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2707.425057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2707.426972] Lustre: Skipped 2 previous similar messages [ 2712.544650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2712.547014] Lustre: Skipped 1 previous similar message [ 2715.103121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2715.153516] Lustre: server umount lustre-OST0000 complete [ 2716.371119] LustreError: 156725:0:(ldlm_lib.c:2986:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2716.374654] Lustre: 156090:0:(ldlm_lib.c:2389:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2716.377186] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2716.383428] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2716.387434] LustreError: 156090:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d3f048cd x1864276454179200/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2716.392544] LustreError: 156090:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2716.395392] LustreError: 156090:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2716.398542] Lustre: lustre-MDT0000: Recovery over after 0:23, of 2 clients 0 recovered and 2 were evicted. [ 2716.503671] Lustre: server umount lustre-MDT0000 complete [ 2717.664123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2717.667405] Lustre: Skipped 1 previous similar message [ 2717.668973] LustreError: 153349:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.693849] LustreError: 154970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912635 with bad export cookie 18416820932562160443 [ 2717.696645] LustreError: 152666:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2717.696894] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2717.699136] LustreError: 154970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2723.882588] Lustre: server umount lustre-MDT0001 complete [ 2727.441080] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2728.333582] Key type lgssc unregistered [ 2728.463409] LNet: 157460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2728.466727] LNetError: 157460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2728.474273] LNet: Removed LNI 192.168.202.108@tcp [ 2728.769512] Key type .llcrypt unregistered [ 2728.770607] Key type ._llcrypt unregistered [ 2737.148216] Key type ._llcrypt registered [ 2737.149290] Key type .llcrypt registered [ 2737.183914] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2737.558899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2737.583034] alg: No test for adler32 (adler32-zlib) [ 2738.429614] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 2738.510426] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2740.087095] Key type lgssc registered [ 2740.446846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2743.030186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2743.037118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.185513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.418963] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2748.119578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.239869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2749.487753] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2751.267778] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2752.438235] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2754.554846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2754.670714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2756.666530] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2758.431517] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2770.022353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2775.521182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.521966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2775.522830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2775.524543] Lustre: Skipped 1 previous similar message [ 2780.641909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2780.643766] Lustre: Skipped 2 previous similar messages [ 2781.390720] Lustre: server umount lustre-OST0000 complete [ 2784.223963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2784.228344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.236838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2785.760966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.695441] Lustre: server umount lustre-MDT0000 complete [ 2789.893466] LustreError: 160445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912707 with bad export cookie 5440611580041139806 [ 2789.896501] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2789.898975] LustreError: 160445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2789.999329] Lustre: server umount lustre-MDT0001 complete [ 2793.409031] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2794.368250] Key type lgssc unregistered [ 2794.489346] LNet: 162024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2794.491835] LNetError: 162024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2794.500329] LNet: Removed LNI 192.168.202.108@tcp [ 2794.792308] Key type .llcrypt unregistered [ 2794.793390] Key type ._llcrypt unregistered [ 2806.214810] Key type ._llcrypt registered [ 2806.216159] Key type .llcrypt registered [ 2806.253491] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2806.665228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2806.671843] alg: No test for adler32 (adler32-zlib) [ 2807.530428] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 2807.612564] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2809.191177] Key type lgssc registered [ 2809.565052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2812.144878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2812.150642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.301397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2814.602908] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2817.378862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.507897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2818.755754] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2820.542974] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2821.650286] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2823.758467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2823.862751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.669408] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2827.320433] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2838.182305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2847.711846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2847.714154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.718101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2848.737326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2848.737646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2848.737769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.739689] Lustre: Skipped 1 previous similar message [ 2849.992524] Lustre: server umount lustre-OST0000 complete [ 2853.343829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2853.345876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.349118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2857.235805] Lustre: server umount lustre-MDT0000 complete [ 2858.390088] LustreError: 163665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912775 with bad export cookie 8842755084103058487 [ 2858.393094] LustreError: 163665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2858.393499] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2858.506837] Lustre: server umount lustre-MDT0001 complete [ 2861.864979] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2862.795164] Key type lgssc unregistered [ 2862.912338] LNet: 166878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2862.915537] LNetError: 166878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2862.925312] LNet: Removed LNI 192.168.202.108@tcp [ 2863.192105] Key type .llcrypt unregistered [ 2863.193097] Key type ._llcrypt unregistered [ 2871.254300] Key type ._llcrypt registered [ 2871.255329] Key type .llcrypt registered [ 2871.290706] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2871.578630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2871.586714] alg: No test for adler32 (adler32-zlib) [ 2872.435345] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 2872.517678] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2874.103238] Key type lgssc registered [ 2874.469555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2878.009556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2878.015832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.149396] Lustre: *** cfs_fail_loc=135, val=0*** [ 2879.189334] LustreError: 168362:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2879.191942] LustreError: 168362:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.108@tcp: cfg command failed: rc = -2 [ 2879.195793] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2879.195793] [ 2879.198585] LustreError: MGC192.168.202.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2879.203480] LustreError: 168323:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2879.205993] LustreError: 168323:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2879.208057] LustreError: 168323:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2879.209665] LustreError: 168323:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2879.242940] Lustre: server umount lustre-MDT0000 complete [ 2879.244198] LustreError: 168323:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2881.385075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.531613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.743932] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2885.476130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.596054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2886.830614] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2888.498842] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2889.687326] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2890.330258] LustreError: 168764:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2890.333568] LustreError: 168764:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2890.335988] LustreError: 168764:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 2890.720426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2890.720645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2890.721171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2890.728675] Lustre: Skipped 1 previous similar message [ 2895.840116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.842331] Lustre: Skipped 2 previous similar messages [ 2896.466583] Lustre: server umount lustre-MDT0000 complete [ 2897.677117] LustreError: 168717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912815 with bad export cookie 11359383571691639796 [ 2897.680563] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2897.786956] Lustre: server umount lustre-MDT0001 complete [ 2901.472369] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2902.566875] Key type lgssc unregistered [ 2902.689397] LNet: 170809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2902.691819] LNetError: 170809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2902.701427] LNet: Removed LNI 192.168.202.108@tcp [ 2903.020420] Key type .llcrypt unregistered [ 2903.021964] Key type ._llcrypt unregistered [ 2909.241322] Key type ._llcrypt registered [ 2909.242152] Key type .llcrypt registered [ 2909.273666] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2909.358215] Key type .llcrypt unregistered [ 2909.359291] Key type ._llcrypt unregistered [ 2916.098127] Key type ._llcrypt registered [ 2916.099195] Key type .llcrypt registered [ 2916.138569] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2916.466148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2916.508115] alg: No test for adler32 (adler32-zlib) [ 2917.353437] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 2917.430390] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2919.007114] Key type lgssc registered [ 2919.356779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2921.739113] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2921.744692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2937.183290] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2942.303198] LustreError: 172883:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 2968.927180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2970.820560] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2972.597772] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2974.889560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.056895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.077273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2976.348387] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2979.161338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.287271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2980.538764] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2982.210654] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2983.338813] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2984.238678] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2994.657525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.659565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2994.661950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2994.666934] Lustre: Skipped 1 previous similar message [ 2998.864386] Lustre: server umount lustre-OST0000 complete [ 3004.896510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3004.896895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.897300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3004.904383] Lustre: Skipped 2 previous similar messages [ 3006.295611] Lustre: server umount lustre-MDT0000 complete [ 3007.491337] LustreError: 175202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777912924 with bad export cookie 12508708403927995837 [ 3007.494754] LustreError: 175202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3007.494808] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.609420] Lustre: server umount lustre-MDT0001 complete [ 3011.318334] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3012.358850] Key type lgssc unregistered [ 3012.489358] LNet: 176584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3012.492137] LNetError: 176584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3012.501480] LNet: Removed LNI 192.168.202.108@tcp [ 3012.844491] Key type .llcrypt unregistered [ 3012.845661] Key type ._llcrypt unregistered [ 3021.722503] Key type ._llcrypt registered [ 3021.723525] Key type .llcrypt registered [ 3021.759486] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3022.178099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3022.183854] alg: No test for adler32 (adler32-zlib) [ 3023.025363] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 3023.109703] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3024.687309] Key type lgssc registered [ 3025.020242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3027.458652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3027.464496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3028.609293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3029.855344] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3032.488491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3032.612957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3033.881300] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3035.638204] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3036.803865] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3038.860981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3038.972390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3040.708720] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3042.409626] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3054.373790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3059.324741] Lustre: Failing over lustre-MDT0000 [ 3059.414947] Lustre: server umount lustre-MDT0000 complete [ 3059.681555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.682609] LustreError: 177951:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.684560] Lustre: Skipped 2 previous similar messages [ 3059.692077] LustreError: 177951:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3063.368558] LustreError: 178666:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.372714] LustreError: 178666:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3064.800048] LustreError: 179324:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3064.806464] LustreError: 179324:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3068.482322] LustreError: 177951:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3072.255854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.297355] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3072.381973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.400835] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3073.602529] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3073.662347] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3075.127863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3077.604438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3077.612732] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3077.629649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3079.590965] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3084.255818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3084.258336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3084.263570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3084.769104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3084.772270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3087.842601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3087.844467] Lustre: Skipped 1 previous similar message [ 3092.962292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3092.963995] Lustre: Skipped 1 previous similar message [ 3097.055089] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3097.100871] Lustre: server umount lustre-OST0000 complete [ 3103.200803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3103.201125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.201717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.208771] Lustre: Skipped 2 previous similar messages [ 3104.470417] Lustre: server umount lustre-MDT0000 complete [ 3105.742313] LustreError: 179575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913023 with bad export cookie 14216557479455372003 [ 3105.745144] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3105.745732] LustreError: 179575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3105.863465] Lustre: server umount lustre-MDT0001 complete [ 3109.609627] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3110.588912] Key type lgssc unregistered [ 3110.709346] LNet: 182479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3110.711538] LNetError: 182479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3110.719322] LNet: Removed LNI 192.168.202.108@tcp [ 3110.997963] Key type .llcrypt unregistered [ 3110.999021] Key type ._llcrypt unregistered [ 3119.635486] Key type ._llcrypt registered [ 3119.636930] Key type .llcrypt registered [ 3119.677702] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3120.133330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3120.140469] alg: No test for adler32 (adler32-zlib) [ 3120.984023] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 3121.058591] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3122.639156] Key type lgssc registered [ 3122.969823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3125.448082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3125.454945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.629321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3127.857816] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3130.602346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3130.729152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3131.979697] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3133.722080] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3134.851773] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3136.982337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3137.097701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3138.934145] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3140.662526] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3152.486220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3157.174382] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3163.678079] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3174.838857] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3182.559894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3182.561927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3182.565246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3183.586397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3183.586653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3183.590894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3183.592495] Lustre: Skipped 1 previous similar message [ 3187.018990] Lustre: server umount lustre-OST0000 complete [ 3188.704157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.704777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.707180] Lustre: Skipped 1 previous similar message [ 3193.824040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3193.826078] Lustre: Skipped 2 previous similar messages [ 3194.324771] Lustre: server umount lustre-MDT0000 complete [ 3195.479158] LustreError: 186070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913112 with bad export cookie 16310246513454654809 [ 3195.481820] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3195.485260] LustreError: 186070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3195.601453] Lustre: server umount lustre-MDT0001 complete [ 3198.971382] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3199.924208] Key type lgssc unregistered [ 3200.044346] LNet: 187238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3200.046301] LNetError: 187238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3200.058308] LNet: Removed LNI 192.168.202.108@tcp [ 3200.353301] Key type .llcrypt unregistered [ 3200.354305] Key type ._llcrypt unregistered [ 3209.131594] Key type ._llcrypt registered [ 3209.132618] Key type .llcrypt registered [ 3209.174360] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3209.585597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3209.591874] alg: No test for adler32 (adler32-zlib) [ 3210.450592] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 3210.539043] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3212.119141] Key type lgssc registered [ 3212.474836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3214.976112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3214.982802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.130556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.357247] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3219.996068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.119073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3221.393080] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3223.106547] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.198022] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3226.275046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3226.393144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3227.429246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3228.126961] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3229.847519] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3232.162074] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3240.969725] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3249.637466] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3259.511779] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3268.576864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3268.582141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.591492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3271.652360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3271.652666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3271.654926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3276.770919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3276.772619] Lustre: Skipped 1 previous similar message [ 3281.375090] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3281.425977] Lustre: server umount lustre-OST0000 complete [ 3287.010405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.010420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3287.011352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3287.017323] Lustre: Skipped 1 previous similar message [ 3287.024014] Lustre: Skipped 2 previous similar messages [ 3292.129730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3292.132977] Lustre: Skipped 1 previous similar message [ 3297.247154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3297.334184] Lustre: server umount lustre-MDT0000 complete [ 3298.740447] LustreError: 189329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913216 with bad export cookie 13010008128105161712 [ 3298.744968] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3298.747834] LustreError: 189329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3298.892282] Lustre: server umount lustre-MDT0001 complete [ 3302.429847] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3303.685783] Key type lgssc unregistered [ 3303.817504] LNet: 193707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3303.821642] LNetError: 193707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3303.829429] LNet: Removed LNI 192.168.202.108@tcp [ 3304.221992] Key type .llcrypt unregistered [ 3304.224533] Key type ._llcrypt unregistered [ 3313.702145] Key type ._llcrypt registered [ 3313.703023] Key type .llcrypt registered [ 3313.738219] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3314.178401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3314.194443] alg: No test for adler32 (adler32-zlib) [ 3315.048655] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 3315.133411] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3316.711116] Key type lgssc registered [ 3317.040745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3319.444041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3319.450208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3320.604954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.847155] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3324.580294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.693767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3326.061144] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3327.880415] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3329.069415] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3331.376561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3331.495140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3333.415491] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3335.225843] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3345.833828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3349.984035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3349.986438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.990545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3351.009138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3351.009812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.016360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3355.678854] Lustre: server umount lustre-OST0000 complete [ 3360.736254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3360.739700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3360.745431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3360.747684] Lustre: Skipped 1 previous similar message [ 3363.094442] Lustre: server umount lustre-MDT0000 complete [ 3364.285343] LustreError: 195057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913281 with bad export cookie 4951669371047504811 [ 3364.288463] LustreError: 195057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3364.288850] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3364.405784] Lustre: server umount lustre-MDT0001 complete [ 3367.578226] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3368.716241] Key type lgssc unregistered [ 3368.850412] LNet: 198424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3368.853737] LNetError: 198424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3368.862395] LNet: Removed LNI 192.168.202.108@tcp [ 3369.162369] Key type .llcrypt unregistered [ 3369.163723] Key type ._llcrypt unregistered [ 3376.227171] Key type ._llcrypt registered [ 3376.228544] Key type .llcrypt registered [ 3376.262912] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3376.695813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3376.711358] alg: No test for adler32 (adler32-zlib) [ 3377.574689] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 3377.660473] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3379.239124] Key type lgssc registered [ 3379.576413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3382.071739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3382.076779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3383.236330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.443507] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3387.311169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.436287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3388.651503] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3390.372389] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3391.448899] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3393.804084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3393.928614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.109926] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3398.332522] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3408.293592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3412.447810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3412.450845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3412.455915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3413.475573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3413.478635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3413.480508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3414.113880] Lustre: server umount lustre-OST0000 complete [ 3417.643283] Lustre: server umount lustre-MDT0000 complete [ 3418.080114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3418.082453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3418.087538] LustreError: 199667:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.091573] LustreError: 199667:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3418.593100] LustreError: 199661:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.966283] LustreError: 201282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913336 with bad export cookie 7386225329079414527 [ 3418.970635] LustreError: 201282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3418.971245] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3419.085789] Lustre: server umount lustre-MDT0001 complete [ 3422.696520] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3423.783963] Key type lgssc unregistered [ 3423.927598] LNet: 203005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3423.932125] LNetError: 203005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3423.938517] LNet: Removed LNI 192.168.202.108@tcp [ 3424.308760] Key type .llcrypt unregistered [ 3424.309781] Key type ._llcrypt unregistered [ 3432.708104] Key type ._llcrypt registered [ 3432.709285] Key type .llcrypt registered [ 3432.757085] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3433.284548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3433.296444] alg: No test for adler32 (adler32-zlib) [ 3434.150918] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 3434.235114] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3435.823138] Key type lgssc registered [ 3436.348294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3438.869442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3438.876094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.023458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3441.323869] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3444.978632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3445.142813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3446.570851] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3449.043128] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3450.650853] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3453.710043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3453.902792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3456.363963] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3458.137409] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3465.189782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3471.464222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3471.497492] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3471.587764] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3473.598336] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3475.469871] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3476.091194] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3486.442092] Lustre: Permanently deactivating lustre-OST0001 [ 3486.446716] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3491.830038] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3495.052113] Lustre: Permanently reactivating lustre-OST0001 [ 3495.056887] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3495.058512] Lustre: Skipped 2 previous similar messages [ 3500.022319] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3500.028209] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3500.030390] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3500.035769] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3500.047977] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3502.561943] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3502.563842] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3502.565449] Lustre: Skipped 2 previous similar messages [ 3507.683248] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3507.685327] Lustre: Skipped 2 previous similar messages [ 3508.220934] Lustre: server umount lustre-OST0001 complete [ 3511.264724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3511.264836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3511.269727] LustreError: Skipped 1 previous similar message [ 3511.279540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3511.285858] Lustre: Skipped 1 previous similar message [ 3512.806714] LustreError: 205822:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3512.818168] LustreError: 205822:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3515.546683] Lustre: server umount lustre-OST0000 complete [ 3517.921522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3517.922808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3517.927218] Lustre: Skipped 2 previous similar messages [ 3517.932162] Lustre: Skipped 3 previous similar messages [ 3523.040608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3523.044070] Lustre: Skipped 1 previous similar message [ 3531.231169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3531.315826] Lustre: server umount lustre-MDT0000 complete [ 3532.915822] LustreError: 204212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913450 with bad export cookie 2059962560660260569 [ 3532.922704] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3532.926209] LustreError: 204212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3533.071033] Lustre: server umount lustre-MDT0001 complete [ 3537.131240] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3538.206383] Key type lgssc unregistered [ 3538.353613] LNet: 208802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3538.356785] LNetError: 208802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3538.366461] LNet: Removed LNI 192.168.202.108@tcp [ 3538.718359] Key type .llcrypt unregistered [ 3538.719724] Key type ._llcrypt unregistered [ 3554.650956] Key type ._llcrypt registered [ 3554.652519] Key type .llcrypt registered [ 3554.705507] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3555.046379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3555.100466] alg: No test for adler32 (adler32-zlib) [ 3555.942898] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 3556.019171] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3557.607215] Key type lgssc registered [ 3558.088673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3561.068215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3561.078405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.183484] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3562.193031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3562.278480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.117677] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3567.777983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3567.814564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3567.830492] Lustre: 210963:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3567.965052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3569.552139] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3571.390584] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3572.452363] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3574.630687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3574.707103] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3574.764959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.488199] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3578.132799] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3581.462904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3585.096982] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3587.553380] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3593.069069] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3600.850758] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3609.629698] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3626.157039] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3626.161870] Lustre: Skipped 1 previous similar message [ 3658.989050] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3658.991452] Lustre: Skipped 3 previous similar messages [ 3730.257127] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3730.260315] Lustre: Skipped 9 previous similar messages [ 3740.130548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3740.130815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3740.132557] LustreError: Skipped 1 previous similar message [ 3740.133025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3740.138957] Lustre: Skipped 1 previous similar message [ 3745.256049] Lustre: server umount lustre-OST0000 complete [ 3747.295867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3747.299147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3747.304587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3747.307161] Lustre: Skipped 1 previous similar message [ 3750.368627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3750.369330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3755.489775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3755.493449] Lustre: Skipped 2 previous similar messages [ 3759.585775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3759.589061] Lustre: Skipped 1 previous similar message [ 3761.119125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3761.229680] Lustre: server umount lustre-MDT0000 complete [ 3762.815542] LustreError: 210958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913680 with bad export cookie 12654532961322766480 [ 3762.822151] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3762.976641] Lustre: server umount lustre-MDT0001 complete [ 3767.618238] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3769.154729] Key type lgssc unregistered [ 3769.333592] LNet: 215407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3769.339450] LNetError: 215407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3769.349761] LNet: Removed LNI 192.168.202.108@tcp [ 3769.854478] Key type .llcrypt unregistered [ 3769.856878] Key type ._llcrypt unregistered [ 3781.331197] Key type ._llcrypt registered [ 3781.332737] Key type .llcrypt registered [ 3781.400282] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3781.967697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3781.981195] alg: No test for adler32 (adler32-zlib) [ 3782.843728] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 3782.940581] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3784.527171] Key type lgssc registered [ 3785.135723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3789.021796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3789.032243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.210355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3791.743414] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3795.760847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.931809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3797.782925] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3800.025082] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3801.469100] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3804.301891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3804.515523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.858391] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3809.219452] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3814.765093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3830.244443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3830.244664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3830.247630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3830.264071] Lustre: Skipped 1 previous similar message [ 3834.140399] Lustre: server umount lustre-OST0000 complete [ 3836.896577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3836.899528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.905714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3836.910575] Lustre: Skipped 1 previous similar message [ 3840.481032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3840.482359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3845.600861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3845.602486] Lustre: Skipped 2 previous similar messages [ 3849.697516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3849.701388] Lustre: Skipped 1 previous similar message [ 3850.207150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3850.308542] Lustre: server umount lustre-MDT0000 complete [ 3852.081848] LustreError: 219040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777913769 with bad export cookie 10656129054046797125 [ 3852.086825] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3852.089115] LustreError: 219040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3852.233476] Lustre: server umount lustre-MDT0001 complete [ 3856.923977] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3858.344257] Key type lgssc unregistered [ 3858.537618] LNet: 220131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3858.542848] LNetError: 220131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3858.554573] LNet: Removed LNI 192.168.202.108@tcp [ 3859.021364] Key type .llcrypt unregistered [ 3859.023191] Key type ._llcrypt unregistered [ 3870.204340] Key type ._llcrypt registered [ 3870.206272] Key type .llcrypt registered [ 3870.275833] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3870.731561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3870.791629] alg: No test for adler32 (adler32-zlib) [ 3871.665139] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 3871.780446] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3873.375264] Key type lgssc registered [ 3873.965096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4137.655240] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 4139.181967] Key type lgssc unregistered [ 4139.334432] LNet: 222097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4139.336725] LNetError: 222097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4139.348354] LNet: Removed LNI 192.168.202.108@tcp [ 4139.712623] Key type .llcrypt unregistered [ 4139.713844] Key type ._llcrypt unregistered [ 4150.104345] Key type ._llcrypt registered [ 4150.105838] Key type .llcrypt registered [ 4150.167384] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 4158.605454] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 4159.040298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4159.076034] alg: No test for adler32 (adler32-zlib) [ 4159.933258] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 4160.022408] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 4161.607319] Key type lgssc registered [ 4162.238425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4177.987868] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3983 sec ========== 13:01:34 (1777914094) [ 4178.812086] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:01:35 (1777914095) === [ 4180.155847] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:01:36 (1777914096) === [ 4193.524083] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 4194.560826] Key type lgssc unregistered [ 4194.684494] LNet: 227263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4194.688701] LNetError: 227263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4194.696433] LNet: Removed LNI 192.168.202.108@tcp [ 4194.971405] Key type .llcrypt unregistered [ 4194.972538] Key type ._llcrypt unregistered