[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476278353 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002235] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009118] LSM: Security Framework initializing [ 0.010051] Yama: becoming mindful. [ 0.011036] SELinux: Initializing. [ 0.012064] *** VALIDATE selinux *** [ 0.019437] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023396] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026115] *** VALIDATE tmpfs *** [ 0.028075] *** VALIDATE proc *** [ 0.029204] *** VALIDATE cgroup *** [ 0.030009] *** VALIDATE cgroup2 *** [ 0.031235] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032132] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034029] Spectre V2 : User space: Vulnerable [ 0.035010] Speculative Store Bypass: Vulnerable [ 0.038000] debug: unmapping init [mem 0xffffffffaaa59000-0xffffffffaaa60fff] [ 0.039145] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040613] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041026] ... version: 2 [ 0.042019] ... bit width: 48 [ 0.043013] ... generic registers: 4 [ 0.044010] ... value mask: 0000ffffffffffff [ 0.045012] ... max period: 00007fffffffffff [ 0.046016] ... fixed-purpose events: 3 [ 0.047014] ... event mask: 000000070000000f [ 0.048276] rcu: Hierarchical SRCU implementation. [ 0.050509] smp: Bringing up secondary CPUs ... [ 0.051582] x86: Booting SMP configuration: [ 0.052022] .... node #0, CPUs: #1 #2 #3 [ 0.056011] smp: Brought up 1 node, 4 CPUs [ 0.058015] smpboot: Max logical packages: 1 [ 0.059012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136667] node 0 deferred pages initialised in 75ms [ 0.138166] devtmpfs: initialized [ 0.139279] x86/mm: Memory block size: 128MB [ 0.143224] gcov: version magic: 0x41383552 [ 0.144750] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150416] pinctrl core: initialized pinctrl subsystem [ 0.153433] [ 0.153993] ************************************************************* [ 0.157039] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159011] ** ** [ 0.162014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165013] ** ** [ 0.168014] ** This means that this kernel is built to expose internal ** [ 0.170025] ** IOMMU data structures, which may compromise security on ** [ 0.172049] ** your system. ** [ 0.174012] ** ** [ 0.177011] ** If you see this message and you are not debugging the ** [ 0.179010] ** kernel, report this immediately to your vendor! ** [ 0.181008] ** ** [ 0.183011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185011] ************************************************************* [ 0.187597] NET: Registered protocol family 16 [ 0.188423] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.195036] cpuidle: using governor menu [ 0.197082] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198394] PCI: Using configuration type 1 for base access [ 0.201144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218055] cryptd: max_cpu_qlen set to 1000 [ 0.222237] ACPI: Added _OSI(Module Device) [ 0.223010] ACPI: Added _OSI(Processor Device) [ 0.224009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.231463] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237390] ACPI: Interpreter enabled [ 0.239049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.240008] ACPI: Using IOAPIC for interrupt routing [ 0.242109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245467] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257163] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259065] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.265104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270208] acpiphp: Slot [2] registered [ 0.271096] acpiphp: Slot [5] registered [ 0.272100] acpiphp: Slot [6] registered [ 0.274108] acpiphp: Slot [7] registered [ 0.275090] acpiphp: Slot [8] registered [ 0.276089] acpiphp: Slot [9] registered [ 0.278099] acpiphp: Slot [10] registered [ 0.279114] acpiphp: Slot [3] registered [ 0.280089] acpiphp: Slot [4] registered [ 0.282115] acpiphp: Slot [11] registered [ 0.284104] acpiphp: Slot [12] registered [ 0.285171] acpiphp: Slot [13] registered [ 0.286093] acpiphp: Slot [14] registered [ 0.288102] acpiphp: Slot [15] registered [ 0.290109] acpiphp: Slot [16] registered [ 0.291073] acpiphp: Slot [17] registered [ 0.293085] acpiphp: Slot [18] registered [ 0.294110] acpiphp: Slot [19] registered [ 0.296137] acpiphp: Slot [20] registered [ 0.297089] acpiphp: Slot [21] registered [ 0.298089] acpiphp: Slot [22] registered [ 0.300115] acpiphp: Slot [23] registered [ 0.301091] acpiphp: Slot [24] registered [ 0.303109] acpiphp: Slot [25] registered [ 0.304074] acpiphp: Slot [26] registered [ 0.306207] acpiphp: Slot [27] registered [ 0.308123] acpiphp: Slot [28] registered [ 0.309130] acpiphp: Slot [29] registered [ 0.311108] acpiphp: Slot [30] registered [ 0.312113] acpiphp: Slot [31] registered [ 0.314061] PCI host bridge to bus 0000:00 [ 0.315019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.320030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.322025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.325028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.328030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329222] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.331778] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.335274] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.342613] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.347014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355393] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359848] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365986] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.372015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.386116] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.392015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.398432] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.405017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.414019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.429016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.440229] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.449023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.455015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.473018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.487479] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.493021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.499014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.520016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.528813] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.535015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.540015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.553018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.564093] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.569013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.574015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.589014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.596363] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.602023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.608167] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.627021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.637051] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.639370] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.643390] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.646433] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.648175] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.653166] iommu: Default domain type: Passthrough [ 0.654000] SCSI subsystem initialized [ 0.655146] ACPI: bus type USB registered [ 0.657117] usbcore: registered new interface driver usbfs [ 0.659077] usbcore: registered new interface driver hub [ 0.661225] usbcore: registered new device driver usb [ 0.663170] pps_core: LinuxPPS API ver. 1 registered [ 0.665010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.668066] PTP clock support registered [ 0.670427] EDAC MC: Ver: 3.0.0 [ 0.672442] PCI: Using ACPI for IRQ routing [ 0.674407] NetLabel: Initializing [ 0.675009] NetLabel: domain hash size = 128 [ 0.675891] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.677080] NetLabel: unlabeled traffic allowed by default [ 0.679124] vgaarb: loaded [ 0.680240] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.681014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.687000] clocksource: Switched to clocksource kvm-clock [ 0.802931] VFS: Disk quotas dquot_6.6.0 [ 0.804331] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.806635] *** VALIDATE ramfs *** [ 0.807695] *** VALIDATE hugetlbfs *** [ 0.809070] pnp: PnP ACPI init [ 0.810981] pnp: PnP ACPI: found 6 devices [ 0.828911] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.832155] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.835123] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.837912] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.841365] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.844288] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.848341] NET: Registered protocol family 2 [ 0.851399] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.858167] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.860627] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.864828] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.867367] TCP: Hash tables configured (established 65536 bind 65536) [ 0.869918] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.874397] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.876834] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880395] NET: Registered protocol family 1 [ 0.883151] RPC: Registered named UNIX socket transport module. [ 0.885728] RPC: Registered udp transport module. [ 0.887760] RPC: Registered tcp transport module. [ 0.889855] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.892747] NET: Registered protocol family 44 [ 0.894760] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.896479] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.898740] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.901251] PCI: CLS 0 bytes, default 64 [ 0.903160] Unpacking initramfs... [ 2.337743] debug: unmapping init [mem 0xffff8e20bcc54000-0xffff8e20bffbffff] [ 2.341209] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.343053] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.345398] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.831518] Initialise system trusted keyrings [ 2.833117] Key type blacklist registered [ 2.835316] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.868503] zbud: loaded [ 2.884636] *** VALIDATE nfs *** [ 2.887099] *** VALIDATE nfs4 *** [ 2.894574] pstore: using deflate compression [ 2.903106] Platform Keyring initialized [ 3.014943] NET: Registered protocol family 38 [ 3.018336] Key type asymmetric registered [ 3.019362] Asymmetric key parser 'x509' registered [ 3.020777] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.024132] io scheduler mq-deadline registered [ 3.025829] io scheduler kyber registered [ 3.027467] io scheduler bfq registered [ 3.029202] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.031512] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.033629] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.035709] ACPI: Power Button [PWRF] [ 3.040620] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.046469] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.056461] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.062989] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.074454] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.099454] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.126151] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.130158] Non-volatile memory driver v1.3 [ 3.131480] Linux agpgart interface v0.103 [ 3.159171] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.162571] vda: detected capacity change from 0 to 74702848 [ 3.177196] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.180331] vdb: detected capacity change from 0 to 1073741824 [ 3.195708] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.199135] vdc: detected capacity change from 0 to 2621440000 [ 3.212058] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.214172] vdd: detected capacity change from 0 to 2621440000 [ 3.233815] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.237257] vde: detected capacity change from 0 to 4294967296 [ 3.258408] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.261793] vdf: detected capacity change from 0 to 4294967296 [ 3.269314] libphy: Fixed MDIO Bus: probed [ 3.284380] usbcore: registered new interface driver usbserial_generic [ 3.289314] usbserial: USB Serial support registered for generic [ 3.293472] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.300931] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.304258] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.308236] mousedev: PS/2 mouse device common for all mice [ 3.311945] rtc_cmos 00:05: RTC can wake from S4 [ 3.316508] rtc_cmos 00:05: registered as rtc0 [ 3.316601] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.318161] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.324767] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.325199] intel_pstate: CPU model not supported [ 3.330561] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.335387] hid: raw HID events driver (C) Jiri Kosina [ 3.340043] usbcore: registered new interface driver usbhid [ 3.342100] usbhid: USB HID core driver [ 3.343712] drop_monitor: Initializing network drop monitor service [ 3.346048] Initializing XFRM netlink socket [ 3.347944] NET: Registered protocol family 10 [ 3.350936] Segment Routing with IPv6 [ 3.352141] NET: Registered protocol family 17 [ 3.354246] mpls_gso: MPLS GSO support [ 3.359367] RAS: Correctable Errors collector initialized. [ 3.361330] AVX version of gcm_enc/dec engaged. [ 3.362776] AES CTR mode by8 optimization enabled [ 3.451087] sched_clock: Marking stable (3451035079, 0)->(4365610089, -914575010) [ 3.454672] registered taskstats version 1 [ 3.456588] Loading compiled-in X.509 certificates [ 3.458389] zswap: loaded using pool lzo/zbud [ 3.484957] Key type big_key registered [ 3.496393] Key type encrypted registered [ 3.499462] ima: No TPM chip found, activating TPM-bypass! [ 3.501896] ima: Allocated hash algorithm: sha1 [ 3.503875] ima: No architecture policies found [ 3.505867] evm: Initialising EVM extended attributes: [ 3.507820] evm: security.selinux [ 3.509147] evm: security.ima [ 3.509990] evm: security.capability [ 3.511117] evm: HMAC attrs: 0x1 [ 3.513671] rtc_cmos 00:05: setting system clock to 2026-08-19 05:30:26 UTC (1787117426) [ 3.522400] debug: unmapping init [mem 0xffffffffaba03000-0xffffffffabbfffff] [ 3.525930] debug: unmapping init [mem 0xffffffffaa782000-0xffffffffaaa58fff] [ 3.535183] Write protecting the kernel read-only data: 28672k [ 3.537820] debug: unmapping init [mem 0xffffffffa8e03000-0xffffffffa8ffffff] [ 3.540684] debug: unmapping init [mem 0xffffffffa9714000-0xffffffffa97fffff] [ 3.579471] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.588044] systemd[1]: Detected virtualization kvm. [ 3.589396] systemd[1]: Detected architecture x86-64. [ 3.590593] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.615957] systemd[1]: No hostname configured. [ 3.617080] systemd[1]: Set hostname to . [ 3.618814] random: systemd: uninitialized urandom read (16 bytes read) [ 3.620480] systemd[1]: Initializing machine ID from random generator. [ 3.680028] random: ln: uninitialized urandom read (6 bytes read) [ 3.756731] random: systemd: uninitialized urandom read (16 bytes read) [ 3.759092] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.764064] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.773128] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.383092] device-mapper: uevent: version 1.0.3 [ 4.385244] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] [ 5.011606] virtio_net virtio0 ens2: renamed from eth0 Reached target Basic System. [ 5.128450] random: fast init done [ OK ] Started Hard[ 5.150142] scsi host0: ata_piix ware RNG Entropy Gatherer Daemon. [ 5.242456] scsi host1: ata_piix [ 5.243975] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.245988] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.966350] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.028477] random: crng init done [ 10.029813] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.234421] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.739611] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.414489] SELinux: Disabled at runtime. [ 14.525251] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.537918] systemd[1]: Detected virtualization kvm. [ 14.541699] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.260681] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.275830] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.311031] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.332920] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.342628] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.359952] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.389967] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ 16.957820] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 17.585811] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.570147] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.592955] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.073531] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.150401] EDAC sbridge: Ver: 1.1.2 [ 23.194845] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 24.122072] NFS: Registering the id_resolver key type [ 24.131086] Key type id_resolver registered [ 24.136795] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg203-server login: [ 84.451796] libcfs: loading out-of-tree module taints kernel. [ 84.583648] Key type ._llcrypt registered [ 84.588139] Key type .llcrypt registered [ 84.889399] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 107.430680] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 108.984851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 109.003969] alg: No test for adler32 (adler32-zlib) [ 110.612890] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 111.523642] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 113.383223] Key type lgssc registered [ 115.103475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.872153] hrtimer: interrupt took 4998364 ns [ 129.221684] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 168.577707] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 183.091043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 183.145432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.368484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.401947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 184.508839] Lustre: lustre-MDT0000: new disk, initializing [ 184.604106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.624703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.068586] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.983854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.169909] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 204.215849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 204.220554] Lustre: Skipped 1 previous similar message [ 204.350194] Lustre: lustre-MDT0001: new disk, initializing [ 204.456376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.520364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 204.538099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 210.058694] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.400879] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 228.785861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.358714] Lustre: lustre-OST0000: new disk, initializing [ 229.366111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 229.376770] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.524473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.590970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.603899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.748119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.898416] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.687604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.847750] Lustre: lustre-OST0001: new disk, initializing [ 253.860756] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 253.866808] Lustre: 9524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 253.945763] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 260.197239] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.229086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 263.244316] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 263.310474] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 275.671137] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 282.732645] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 289.949058] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing check_logdir /tmp/testlogs/ [ 296.817093] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing yml_node [ 303.310567] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 306.310610] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 310.939829] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 313.360886] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 01:35:34 EDT 2026 [ 339.160518] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 340.953795] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 358.369575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 358.383390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.402919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.416913] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.417247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.450489] Lustre: Skipped 1 previous similar message [ 360.484076] Lustre: Skipped 3 previous similar messages [ 363.603490] Lustre: server umount lustre-MDT0000 complete [ 369.485142] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117792 with bad export cookie 5442129411207124860 [ 369.502128] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.507875] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 370.660041] LustreError: 9384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.665186] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.684176] LustreError: 9384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 370.704357] Lustre: Skipped 1 previous similar message [ 370.713499] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 370.724032] Lustre: Skipped 1 previous similar message [ 374.690717] LustreError: 9384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.698135] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.709494] LustreError: 9384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 374.735267] Lustre: Skipped 1 previous similar message [ 376.240662] Lustre: server umount lustre-MDT0001 complete [ 380.719898] Lustre: server umount lustre-OST0000 complete [ 384.748479] Lustre: server umount lustre-OST0001 complete [ 392.869643] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 405.363822] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 444.219703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.575779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 444.597072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 444.711368] Lustre: lustre-MDT0000: new disk, initializing [ 444.873843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.920859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 450.166329] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 462.610816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.758277] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 462.843710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 462.851432] Lustre: Skipped 1 previous similar message [ 462.996850] Lustre: lustre-MDT0001: new disk, initializing [ 463.089662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 463.148164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 463.162268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 467.550811] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 477.900465] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 486.633639] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 495.313695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 495.625737] Lustre: lustre-OST0000: new disk, initializing [ 495.629772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 495.635323] Lustre: 17054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 495.698371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.754130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 496.775544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 496.934938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 502.078952] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 513.295326] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 519.411733] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 519.643098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 524.877544] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 525.147876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 527.847373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 527.854520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.863105] LustreError: Skipped 1 previous similar message [ 527.879535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 527.887383] Lustre: Skipped 2 previous similar messages [ 533.609618] Lustre: server umount lustre-OST0000 complete [ 538.082068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.091677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 538.097527] Lustre: Skipped 1 previous similar message [ 538.108887] Lustre: Skipped 4 previous similar messages [ 544.139197] Lustre: server umount lustre-MDT0000 complete [ 548.136096] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117971 with bad export cookie 5442129411207126393 [ 548.147170] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.158841] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 548.513862] Lustre: server umount lustre-MDT0001 complete [ 552.178815] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:39:34 (1787117974) [ 559.065079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.583264] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.669335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.569633] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 564.708178] LustreError: 19318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.840632] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.159960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.146673] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 587.690867] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 594.267695] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 603.534362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 611.372916] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 620.669543] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 629.735436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 629.743754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.758727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 629.774691] Lustre: Skipped 1 previous similar message [ 629.789655] Lustre: Skipped 3 previous similar messages [ 635.835238] Lustre: server umount lustre-OST0000 complete [ 641.001152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 646.780548] Lustre: server umount lustre-MDT0000 complete [ 650.216720] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.232803] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 652.683462] LustreError: 19300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118075 with bad export cookie 5442129411207127275 [ 652.701938] LustreError: 19300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 652.707489] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.361889] Lustre: server umount lustre-MDT0001 complete [ 667.771121] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 671.318477] Key type lgssc unregistered [ 671.807232] LNet: 22856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.820283] LNetError: 22856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.856558] LNet: Removed LNI 192.168.202.103@tcp [ 673.588060] Key type .llcrypt unregistered [ 673.592662] Key type ._llcrypt unregistered [ 700.019940] Key type ._llcrypt registered [ 700.024298] Key type .llcrypt registered [ 700.118391] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 701.327851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 701.375727] alg: No test for adler32 (adler32-zlib) [ 702.362757] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 702.583473] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 704.319341] Key type lgssc registered [ 705.914381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 713.321918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 713.337565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.741053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.921249] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 724.962546] LustreError: 24272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 725.004121] LustreError: 24272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 730.087584] LustreError: 24271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 730.120759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.647241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 734.958828] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 745.208084] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 751.651142] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 759.525588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 760.106291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.539166] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 779.536854] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 784.812297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 805.352311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 805.371282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 805.371791] Lustre: Skipped 1 previous similar message [ 805.374192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 805.385835] Lustre: Skipped 1 previous similar message [ 808.067996] Lustre: server umount lustre-OST0000 complete [ 815.587409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 815.609485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 815.624379] Lustre: Skipped 1 previous similar message [ 820.706438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 820.720087] Lustre: Skipped 1 previous similar message [ 825.830094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 825.838203] Lustre: Skipped 1 previous similar message [ 827.359146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 827.592737] Lustre: server umount lustre-MDT0000 complete [ 830.951770] LustreError: 24268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.964543] LustreError: 24268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 831.512720] LustreError: 24252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118254 with bad export cookie 15740999972006757858 [ 831.519194] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 831.526243] LustreError: 24252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 831.808484] Lustre: server umount lustre-MDT0001 complete [ 846.249727] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 848.990519] Key type lgssc unregistered [ 849.451711] LNet: 28253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.461144] LNetError: 28253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.494755] LNet: Removed LNI 192.168.202.103@tcp [ 850.872127] Key type .llcrypt unregistered [ 850.875171] Key type ._llcrypt unregistered [ 880.554415] Key type ._llcrypt registered [ 880.556586] Key type .llcrypt registered [ 880.646955] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 881.957910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.040931] alg: No test for adler32 (adler32-zlib) [ 883.278517] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 883.565714] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 885.338340] Key type lgssc registered [ 886.628459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 896.988626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 897.021713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.572988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 903.928675] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 908.771298] LustreError: 29665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.827375] LustreError: 29665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 913.889731] LustreError: 29666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.011432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.419628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 918.899739] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 927.456916] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 934.802925] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 954.609959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 955.443445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.586699] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 968.773898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 973.607107] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 981.992937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 982.007093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 982.031839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.039536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.041490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.998487] Lustre: server umount lustre-OST0000 complete [ 994.273739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.287956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.293178] Lustre: Skipped 2 previous similar messages [ 998.318896] Lustre: server umount lustre-MDT0000 complete [ 999.395972] LustreError: 29661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.417220] LustreError: 29661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1002.307704] LustreError: 29647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118425 with bad export cookie 14003502590720055318 [ 1002.315364] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.340016] LustreError: 29647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1002.930752] Lustre: server umount lustre-MDT0001 complete [ 1015.858532] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1019.010182] Key type lgssc unregistered [ 1019.373473] LNet: 33551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.384891] LNetError: 33551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.454478] LNet: Removed LNI 192.168.202.103@tcp [ 1021.430401] Key type .llcrypt unregistered [ 1021.436285] Key type ._llcrypt unregistered [ 1049.862314] Key type ._llcrypt registered [ 1049.865613] Key type .llcrypt registered [ 1050.023354] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1051.469166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1051.610280] alg: No test for adler32 (adler32-zlib) [ 1052.820623] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 1053.136435] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1054.879301] Key type lgssc registered [ 1056.261632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1065.879302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1065.909286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.555684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.826934] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1077.733371] LustreError: 34972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.743798] LustreError: 34972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1082.300705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.677277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1087.401582] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1096.137618] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1103.623976] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1110.622260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1111.049648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1112.118720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1118.357637] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1126.539963] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1137.648361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.660290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1137.684278] LustreError: Skipped 1 previous similar message [ 1137.693769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.706293] Lustre: Skipped 1 previous similar message [ 1142.755309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.769773] Lustre: Skipped 1 previous similar message [ 1144.189964] Lustre: server umount lustre-OST0000 complete [ 1149.411629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1149.425886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.445544] Lustre: Skipped 1 previous similar message [ 1149.453808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1152.993833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1152.997415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.112698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1158.122411] Lustre: Skipped 2 previous similar messages [ 1163.231319] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1163.820859] Lustre: server umount lustre-MDT0000 complete [ 1167.329390] LustreError: 35983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.362234] LustreError: 35983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1168.388730] LustreError: 37756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118591 with bad export cookie 14298629536987359625 [ 1168.395273] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1168.406934] LustreError: 37756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1168.937856] Lustre: server umount lustre-MDT0001 complete [ 1184.383816] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1188.159843] Key type lgssc unregistered [ 1188.582529] LNet: 38511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.591188] LNetError: 38511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1188.614212] LNet: Removed LNI 192.168.202.103@tcp [ 1189.375647] Key type .llcrypt unregistered [ 1189.377531] Key type ._llcrypt unregistered [ 1221.775824] Key type ._llcrypt registered [ 1221.781069] Key type .llcrypt registered [ 1221.926549] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1223.362042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1223.424079] alg: No test for adler32 (adler32-zlib) [ 1224.551311] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 1224.870105] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1226.591192] Key type lgssc registered [ 1227.902455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1236.705539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1236.738577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1238.544662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1244.030660] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1248.756628] LustreError: 39927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.792483] LustreError: 39927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1253.860842] LustreError: 39928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.780757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.391821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1262.687908] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1272.310397] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1281.451690] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1291.321465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1292.035913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.042224] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1308.428369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1311.979961] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1320.723832] Lustre: lustre-OST0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1322.978846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1322.995054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.020595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1324.000984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1324.017062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.021207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1329.123310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1329.138305] Lustre: Skipped 2 previous similar messages [ 1334.241441] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1334.258147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1334.281286] Lustre: Skipped 2 previous similar messages [ 1334.522284] Lustre: server umount lustre-OST0000 complete [ 1359.845742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.848044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1359.880701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.901468] Lustre: Skipped 2 previous similar messages [ 1361.971491] Lustre: server umount lustre-MDT0000 complete [ 1364.974138] LustreError: 39922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.009630] LustreError: 39922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1365.613249] LustreError: 39909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118788 with bad export cookie 9932804934418514548 [ 1365.619648] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1365.623346] LustreError: 39909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1372.763922] Lustre: server umount lustre-MDT0001 complete [ 1389.988155] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1392.946961] Key type lgssc unregistered [ 1393.315958] LNet: 43661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.323983] LNetError: 43661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.345539] LNet: Removed LNI 192.168.202.103@tcp [ 1394.329265] Key type .llcrypt unregistered [ 1394.331430] Key type ._llcrypt unregistered [ 1422.128642] Key type ._llcrypt registered [ 1422.132261] Key type .llcrypt registered [ 1422.257669] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1423.708445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1423.754405] alg: No test for adler32 (adler32-zlib) [ 1424.780935] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 1425.029240] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1426.735211] Key type lgssc registered [ 1427.943444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1436.005618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1436.019745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1437.375958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1441.626589] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1447.905209] LustreError: 45074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1447.927120] LustreError: 45074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1449.686870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.928468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1454.178661] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1462.997862] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1470.442454] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1477.207844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1477.487338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.624197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1482.855365] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1489.798736] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1501.153076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1501.176705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.208500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1502.989794] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1505.270113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1505.279362] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1505.291671] Lustre: Skipped 1 previous similar message [ 1506.700727] Lustre: server umount lustre-MDT0000 complete [ 1508.075722] LustreError: 45069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.372687] LustreError: 45074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.399945] LustreError: 45074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1511.193040] LustreError: 45056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118934 with bad export cookie 8008183155057877344 [ 1511.198352] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1511.203496] LustreError: 45056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1513.188604] LustreError: 45070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1513.215680] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.3@tcp (stopping) [ 1513.223254] Lustre: Skipped 2 previous similar messages [ 1515.491811] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.833644] Lustre: server umount lustre-MDT0001 complete [ 1532.210204] Lustre: server umount lustre-OST0000 complete [ 1550.308595] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1553.060836] Key type lgssc unregistered [ 1553.352855] LNet: 48938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.361979] LNetError: 48938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.417823] LNet: Removed LNI 192.168.202.103@tcp [ 1555.233807] Key type .llcrypt unregistered [ 1555.238929] Key type ._llcrypt unregistered [ 1580.599612] Key type ._llcrypt registered [ 1580.601291] Key type .llcrypt registered [ 1580.689212] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1581.552921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1581.641359] alg: No test for adler32 (adler32-zlib) [ 1582.654473] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 1582.854834] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1584.511166] Key type lgssc registered [ 1585.665916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1592.726861] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1592.756471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1608.287433] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1613.407419] LustreError: 50328:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 1640.032171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.247585] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1651.754165] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1820.647840] Lustre: server umount lustre-OST0000 complete [ 1838.332287] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1840.911600] Key type lgssc unregistered [ 1841.134862] LNet: 52017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.140385] LNetError: 52017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.154922] LNet: Removed LNI 192.168.202.103@tcp [ 1841.880237] Key type .llcrypt unregistered [ 1841.882373] Key type ._llcrypt unregistered [ 1868.269348] Key type ._llcrypt registered [ 1868.272464] Key type .llcrypt registered [ 1868.406041] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1869.537523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1869.621294] alg: No test for adler32 (adler32-zlib) [ 1870.780658] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 1871.032371] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1872.743200] Key type lgssc registered [ 1874.081972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1881.873882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1881.889813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.383983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.129325] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1893.860078] LustreError: 53436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1893.899092] LustreError: 53436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1898.979262] LustreError: 53437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.049338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.413727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1905.596638] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1913.087525] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1918.987400] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1926.434107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1926.709371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.757883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1934.308434] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1943.204110] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1949.152841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1949.162557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.180648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1950.183479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1950.197976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1950.204814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.237235] Lustre: Skipped 1 previous similar message [ 1952.220424] Lustre: server umount lustre-OST0000 complete [ 1956.835085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1956.837951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.849506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1960.419777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1960.426724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.419916] Lustre: server umount lustre-MDT0000 complete [ 1965.539885] LustreError: 53436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1965.559094] LustreError: 53436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1966.557826] LustreError: 55278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119389 with bad export cookie 11817090923789853851 [ 1966.560711] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1966.570339] LustreError: 55278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1967.112363] Lustre: server umount lustre-MDT0001 complete [ 1980.735223] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1984.380941] Key type lgssc unregistered [ 1984.896871] LNet: 56972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.904739] LNetError: 56972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.923209] LNet: Removed LNI 192.168.202.103@tcp [ 1986.000422] Key type .llcrypt unregistered [ 1986.003626] Key type ._llcrypt unregistered [ 2009.596401] Key type ._llcrypt registered [ 2009.599357] Key type .llcrypt registered [ 2009.731314] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2010.933979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2010.986784] alg: No test for adler32 (adler32-zlib) [ 2012.036351] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 2012.243907] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2013.929444] Key type lgssc registered [ 2014.896888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2022.349590] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2022.369987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2037.920177] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2043.041461] LustreError: 58359:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 2069.727264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.313877] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2082.724673] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2089.846299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.322533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.387643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2094.706935] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2095.587828] LustreError: 58385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2095.610096] LustreError: 58385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2100.707193] LustreError: 58384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.836169] LustreError: 59395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.851241] LustreError: 59395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2107.140833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.667632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2113.611136] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2122.447782] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2128.948292] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2131.430047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2131.440583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.467898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2133.474372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2133.485791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2133.494354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.530706] Lustre: Skipped 1 previous similar message [ 2138.593081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2138.598124] Lustre: Skipped 1 previous similar message [ 2143.729736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2143.740271] Lustre: Skipped 1 previous similar message [ 2145.247230] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2145.408273] Lustre: server umount lustre-OST0000 complete [ 2169.315154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2169.325843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.332894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.342604] Lustre: Skipped 1 previous similar message [ 2173.844816] Lustre: server umount lustre-MDT0000 complete [ 2177.941940] LustreError: 58366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119600 with bad export cookie 1683799492841036609 [ 2177.949906] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2177.957532] LustreError: 58366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2192.351318] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2192.716577] Lustre: server umount lustre-MDT0001 complete [ 2203.548117] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2205.969769] Key type lgssc unregistered [ 2206.359258] LNet: 62117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.393295] LNetError: 62117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.422307] LNet: Removed LNI 192.168.202.103@tcp [ 2207.558944] Key type .llcrypt unregistered [ 2207.566207] Key type ._llcrypt unregistered [ 2233.864693] Key type ._llcrypt registered [ 2233.867631] Key type .llcrypt registered [ 2234.022738] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2235.913596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2236.001625] alg: No test for adler32 (adler32-zlib) [ 2236.964822] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 2237.141685] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2238.783690] Key type lgssc registered [ 2239.585274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2248.427221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2248.459134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.910776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.857704] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2259.425134] LustreError: 63532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.443687] LustreError: 63532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2262.282534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.653691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2268.524509] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2277.396896] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2283.221913] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2290.024882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2290.410845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2294.524553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2296.325837] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2304.333319] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2323.430419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.433857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.436099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.462564] Lustre: Skipped 1 previous similar message [ 2324.162701] Lustre: server umount lustre-OST0000 complete [ 2328.546219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.547615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2328.558484] Lustre: Skipped 1 previous similar message [ 2328.571654] Lustre: Skipped 2 previous similar messages [ 2333.667664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2333.672961] Lustre: Skipped 1 previous similar message [ 2334.780927] Lustre: server umount lustre-MDT0000 complete [ 2338.475054] LustreError: 63512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119761 with bad export cookie 3493642235803019916 [ 2338.478554] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2338.484468] LustreError: 63512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2338.755772] Lustre: server umount lustre-MDT0001 complete [ 2350.113244] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2353.348904] Key type lgssc unregistered [ 2353.639631] LNet: 67093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2353.648965] LNetError: 67093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2353.677761] LNet: Removed LNI 192.168.202.103@tcp [ 2354.702770] Key type .llcrypt unregistered [ 2354.708206] Key type ._llcrypt unregistered [ 2382.746936] Key type ._llcrypt registered [ 2382.748736] Key type .llcrypt registered [ 2382.892437] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2383.895934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2383.923080] alg: No test for adler32 (adler32-zlib) [ 2384.884665] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 2385.159638] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2386.863412] Key type lgssc registered [ 2387.823783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2396.765567] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 02:10:19 (1787119819) [ 2403.770241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2403.807586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.233992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.513246] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2415.585922] LustreError: 68798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.599478] LustreError: 68798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2416.628210] LustreError: 68799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.864350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.322444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2423.520827] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2431.810812] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2439.139935] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2446.670258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2447.162447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2451.266592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2454.020299] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2461.785652] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2468.744795] Lustre: Failing over lustre-MDT0000 [ 2468.837944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.850409] Lustre: Skipped 1 previous similar message [ 2468.856772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2468.862067] Lustre: Skipped 2 previous similar messages [ 2469.105217] Lustre: server umount lustre-MDT0000 complete [ 2469.636169] LustreError: 68794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2473.952635] LustreError: 68799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2473.987237] LustreError: 68799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2479.074980] LustreError: 68794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.101495] LustreError: 68794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2479.509645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.598863] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2479.738846] Lustre: *** cfs_fail_loc=135, val=0*** [ 2479.831087] LustreError: 71886:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2479.839823] LustreError: 71886:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.103@tcp: cfg command failed: rc = -2 [ 2479.849869] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2479.849869] [ 2479.864943] LustreError: MGC192.168.202.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2479.894438] LustreError: 71874:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2479.917658] LustreError: 71874:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2479.941459] LustreError: 71874:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2479.952869] LustreError: 71874:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2479.984583] Lustre: server umount lustre-MDT0000 complete [ 2479.987932] LustreError: 71874:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2487.284120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.379547] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2487.392297] LustreError: 68791:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2487.667958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.704964] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2490.087797] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2492.644665] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2492.916151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2492.984882] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2493.045199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2503.140892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.143562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2503.158591] Lustre: Skipped 1 previous similar message [ 2507.584496] Lustre: server umount lustre-OST0000 complete [ 2513.380160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.385101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2513.399524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2513.399532] Lustre: Skipped 1 previous similar message [ 2517.937923] Lustre: server umount lustre-MDT0000 complete [ 2518.499982] LustreError: 68795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.535758] LustreError: 68795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2521.601359] LustreError: 69533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119944 with bad export cookie 12115515466412600666 [ 2521.610589] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2521.856976] Lustre: server umount lustre-MDT0001 complete [ 2531.994842] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2535.287080] Key type lgssc unregistered [ 2535.558529] LNet: 73668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2535.571064] LNetError: 73668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2535.593323] LNet: Removed LNI 192.168.202.103@tcp [ 2536.679832] Key type .llcrypt unregistered [ 2536.685232] Key type ._llcrypt unregistered [ 2562.285331] Key type ._llcrypt registered [ 2562.289135] Key type .llcrypt registered [ 2562.459877] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2563.968908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2564.039367] alg: No test for adler32 (adler32-zlib) [ 2565.138455] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 2565.405038] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2567.175888] Key type lgssc registered [ 2568.446418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2575.942098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2575.962763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.544496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.929793] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2587.107911] LustreError: 75082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.150219] LustreError: 75082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2592.226669] LustreError: 75082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.209772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2595.065735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2600.396907] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2608.951385] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2614.852727] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2621.807399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2622.246476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.314403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2628.162309] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2637.857143] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2647.190666] Lustre: Failing over lustre-MDT0000 [ 2647.955537] Lustre: server umount lustre-MDT0000 complete [ 2649.944908] LustreError: 76230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.085674] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.104443] Lustre: Skipped 1 previous similar message [ 2654.954091] LustreError: 76230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2654.969612] LustreError: 76230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2660.072587] LustreError: 75077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.087964] LustreError: 75077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2660.101647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.252207] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2660.475969] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2660.647546] LustreError: 78174:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2660.653585] LustreError: 78174:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.103@tcp: cfg command failed: rc = -9 [ 2660.678584] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2660.678584] [ 2660.693921] LustreError: MGC192.168.202.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2660.720853] LustreError: 78162:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2660.729669] LustreError: 78162:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2660.738700] LustreError: 78162:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2660.753875] LustreError: 78162:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2660.788611] Lustre: server umount lustre-MDT0000 complete [ 2660.796558] LustreError: 78162:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2668.248088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.333605] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2668.346097] LustreError: 78521:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.103@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000002563c4d5) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2668.361704] LustreError: 75075:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2668.618193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.657485] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2670.322642] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2673.209229] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2673.650826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2673.676392] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2673.736769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2683.891447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.918892] Lustre: Skipped 2 previous similar messages [ 2683.941717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2683.966899] Lustre: Skipped 1 previous similar message [ 2688.888051] Lustre: server umount lustre-OST0000 complete [ 2694.115235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2694.121177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.147052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2694.164754] Lustre: Skipped 1 previous similar message [ 2699.236135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.246066] Lustre: Skipped 1 previous similar message [ 2699.821132] Lustre: server umount lustre-MDT0000 complete [ 2703.333457] LustreError: 75082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.346963] LustreError: 75082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 2703.857778] LustreError: 75065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120126 with bad export cookie 3318292305555749827 [ 2703.862457] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2703.887510] LustreError: 75065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2704.366088] Lustre: server umount lustre-MDT0001 complete [ 2716.384733] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2719.490963] Key type lgssc unregistered [ 2719.783827] LNet: 79957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2719.793132] LNetError: 79957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2719.813315] LNet: Removed LNI 192.168.202.103@tcp [ 2720.422245] Key type .llcrypt unregistered [ 2720.425356] Key type ._llcrypt unregistered [ 2738.448442] Key type ._llcrypt registered [ 2738.451370] Key type .llcrypt registered [ 2738.687452] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2738.938453] Key type .llcrypt unregistered [ 2738.942468] Key type ._llcrypt unregistered [ 2769.944265] Key type ._llcrypt registered [ 2769.946647] Key type .llcrypt registered [ 2770.066577] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2771.614891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2771.642587] alg: No test for adler32 (adler32-zlib) [ 2772.652340] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 2772.930439] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2774.687133] Key type lgssc registered [ 2775.591276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2783.134847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2783.150373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.678347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2789.395415] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2794.977819] LustreError: 81865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2794.997333] LustreError: 81865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2797.439172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.711790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2802.418789] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2810.261696] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2816.557953] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2824.259089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2824.721298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2829.877748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2830.614661] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2838.307937] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2852.323913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.327439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2852.338937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.442720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.451398] Lustre: Skipped 1 previous similar message [ 2857.819956] Lustre: server umount lustre-OST0000 complete [ 2862.562391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.582858] Lustre: Skipped 1 previous similar message [ 2862.599345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.605268] Lustre: Skipped 2 previous similar messages [ 2867.683360] LustreError: 81865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.693015] LustreError: 81865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2867.757035] Lustre: server umount lustre-MDT0000 complete [ 2871.211637] LustreError: 84358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120294 with bad export cookie 12339076894669061208 [ 2871.227745] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2871.648879] Lustre: server umount lustre-MDT0001 complete [ 2883.022479] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2885.684587] Key type lgssc unregistered [ 2886.055523] LNet: 85418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2886.070125] LNetError: 85418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2886.088842] LNet: Removed LNI 192.168.202.103@tcp [ 2886.973897] Key type .llcrypt unregistered [ 2886.979794] Key type ._llcrypt unregistered [ 2912.106312] Key type ._llcrypt registered [ 2912.108534] Key type .llcrypt registered [ 2912.200496] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2913.515245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2913.564612] alg: No test for adler32 (adler32-zlib) [ 2914.595990] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 2914.851358] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2916.519427] Key type lgssc registered [ 2917.676938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2925.497477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2925.514453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2926.925753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.722399] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2936.293482] LustreError: 86840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.306175] LustreError: 86840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2937.798051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.144371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2941.984823] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2948.481539] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2953.558465] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2960.300677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2960.662047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.666204] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2971.949692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2972.055051] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2979.298388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2979.310646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2979.322623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2982.371805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2982.372694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2982.398870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.453171] Lustre: Skipped 1 previous similar message [ 2984.454993] Lustre: server umount lustre-OST0000 complete [ 2987.489380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2987.508273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2987.520581] Lustre: Skipped 1 previous similar message [ 2992.609690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2992.617285] Lustre: Skipped 1 previous similar message [ 2993.698522] Lustre: server umount lustre-MDT0000 complete [ 2996.782095] LustreError: 86821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120419 with bad export cookie 14069377475259588835 [ 2996.783239] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2996.791420] LustreError: 86821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2997.040450] Lustre: server umount lustre-MDT0001 complete [ 3006.738389] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3008.849522] Key type lgssc unregistered [ 3009.123970] LNet: 90376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3009.131560] LNetError: 90376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3010.150569] LNet: Removed LNI 192.168.202.103@tcp [ 3010.813348] Key type .llcrypt unregistered [ 3010.815424] Key type ._llcrypt unregistered [ 3032.487719] Key type ._llcrypt registered [ 3032.495327] Key type .llcrypt registered [ 3032.626427] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3033.950662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3033.986184] alg: No test for adler32 (adler32-zlib) [ 3034.989940] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 3035.172724] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3036.791268] Key type lgssc registered [ 3037.875524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3045.065982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3045.091763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.487778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3050.646953] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3056.627349] LustreError: 91787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3056.647504] LustreError: 91787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3058.916554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.457492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3063.440743] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3070.136860] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3075.298195] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3081.529334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3081.870628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.680065] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3091.065481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3094.803471] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3105.766291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3105.775076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.786601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3106.785163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3106.786595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3111.912598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3111.923216] Lustre: Skipped 2 previous similar messages [ 3117.034237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3117.047791] Lustre: Skipped 1 previous similar message [ 3120.095162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3120.302495] Lustre: server umount lustre-OST0000 complete [ 3126.240384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3126.256606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3126.274134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.910599] Lustre: server umount lustre-MDT0000 complete [ 3132.395741] LustreError: 91788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.404322] LustreError: 91788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3134.714445] LustreError: 91769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120557 with bad export cookie 1873132363179421653 [ 3134.716193] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.723691] LustreError: 91769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3135.174309] Lustre: server umount lustre-MDT0001 complete [ 3147.184710] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3150.290967] Key type lgssc unregistered [ 3150.661951] LNet: 95323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3150.669542] LNetError: 95323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3150.718432] LNet: Removed LNI 192.168.202.103@tcp [ 3151.856870] Key type .llcrypt unregistered [ 3151.859529] Key type ._llcrypt unregistered [ 3175.546286] Key type ._llcrypt registered [ 3175.548062] Key type .llcrypt registered [ 3175.646959] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3176.735957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3176.757051] alg: No test for adler32 (adler32-zlib) [ 3177.736210] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 3177.943593] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3179.599202] Key type lgssc registered [ 3180.617712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3187.068104] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3187.092177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3202.592438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3207.711319] LustreError: 96725:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 3234.399877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.433696] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3248.374096] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3255.047843] Lustre: server umount lustre-OST0000 complete [ 3264.461880] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:24:46 (1787120686) [ 3271.132642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.633857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3276.119447] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3276.772551] LustreError: 98365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.786425] LustreError: 98365:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3281.901760] LustreError: 98364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.241508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3284.625700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3289.112114] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3296.278612] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3301.400630] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3307.498831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3307.859306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3313.632531] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3316.103354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3321.046375] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3335.649678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3335.661667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3335.677652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3336.687140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.689442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3336.689885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3336.709026] Lustre: Skipped 1 previous similar message [ 3338.499416] Lustre: server umount lustre-OST0000 complete [ 3345.894766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3345.904190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.922850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.037315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.044740] Lustre: Skipped 3 previous similar messages [ 3356.127102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3356.511758] Lustre: server umount lustre-MDT0000 complete [ 3357.157981] LustreError: 98359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.179765] LustreError: 98359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3360.357556] LustreError: 98344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120783 with bad export cookie 9760809568122749168 [ 3360.358863] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3360.378611] LustreError: 98344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3360.892378] Lustre: server umount lustre-MDT0001 complete [ 3371.741137] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3374.642740] Key type lgssc unregistered [ 3374.942953] LNet: 102062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3374.948103] LNetError: 102062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3374.966291] LNet: Removed LNI 192.168.202.103@tcp [ 3375.665884] Key type .llcrypt unregistered [ 3375.667664] Key type ._llcrypt unregistered [ 3417.482303] Key type ._llcrypt registered [ 3417.485142] Key type .llcrypt registered [ 3417.575707] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3418.941980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3418.965847] alg: No test for adler32 (adler32-zlib) [ 3420.117706] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 3420.470195] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3422.127181] Key type lgssc registered [ 3423.301851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3430.500179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3430.513466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.031360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3435.854727] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3441.127378] LustreError: 104248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.167375] LustreError: 104248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3444.799415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3445.078430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3448.956945] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3455.727892] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3461.115514] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3467.400823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3467.687065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.765572] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3476.931451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3481.525205] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3490.273181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3490.286611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3490.304973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3492.320539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3492.325764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3492.331029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.348848] Lustre: Skipped 1 previous similar message [ 3494.591942] Lustre: server umount lustre-OST0000 complete [ 3501.537561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3501.553789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.585099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3504.896101] Lustre: server umount lustre-MDT0000 complete [ 3507.681185] LustreError: 104243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3507.699743] LustreError: 104243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3508.612394] LustreError: 104228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120931 with bad export cookie 571496863864618821 [ 3508.624051] LustreError: 104228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3508.631317] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3509.124612] Lustre: server umount lustre-MDT0001 complete [ 3523.045631] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3526.142977] Key type lgssc unregistered [ 3526.412309] LNet: 107785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3526.419877] LNetError: 107785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3526.431698] LNet: Removed LNI 192.168.202.103@tcp [ 3527.213509] Key type .llcrypt unregistered [ 3527.217408] Key type ._llcrypt unregistered [ 3547.309329] Key type ._llcrypt registered [ 3547.312627] Key type .llcrypt registered [ 3547.417598] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3548.973564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3549.033713] alg: No test for adler32 (adler32-zlib) [ 3550.211714] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 3550.502040] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3552.191216] Key type lgssc registered [ 3553.221821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3562.005949] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3562.022823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3577.633982] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3582.751341] LustreError: 109035:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 3609.439277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3617.664572] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3628.269552] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3637.496569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3637.902194] LustreError: MGC192.168.202.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3637.922771] LustreError: 110061:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3637.935615] LustreError: 110061:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3637.942982] LustreError: 110061:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3638.032973] Lustre: server umount lustre-MDT0000 complete [ 3638.037856] LustreError: 110061:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3658.207460] Lustre: 108374:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787121065/real 1787121065] req@00000000b7044f5d x1873932161387904/t0(0) o400->MGC192.168.202.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787121081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3658.269884] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3658.294287] LustreError: 109047:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3678.772088] Lustre: server umount lustre-OST0000 complete [ 3690.261333] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 3703.875396] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3742.415951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.859134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3742.930848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3743.063293] Lustre: lustre-MDT0000: new disk, initializing [ 3743.217355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3743.258601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3749.026889] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3761.457699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.700716] Lustre: 112856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3761.810486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3761.826823] Lustre: Skipped 1 previous similar message [ 3761.931727] Lustre: lustre-MDT0001: new disk, initializing [ 3762.081403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3762.188154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3762.210279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3768.572217] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3779.483958] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3785.439502] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3794.294423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3794.705543] Lustre: lustre-OST0000: new disk, initializing [ 3794.714513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3794.730148] Lustre: 114824:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3794.873487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.749741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3801.776683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3801.875528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3803.028322] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3813.422952] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3819.456427] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3819.685743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3824.541314] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3824.775203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3827.694877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.696664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3827.708785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.732449] Lustre: Skipped 1 previous similar message [ 3827.771546] LustreError: Skipped 1 previous similar message [ 3832.976280] Lustre: server umount lustre-OST0000 complete [ 3837.921437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.923713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3837.935581] Lustre: Skipped 1 previous similar message [ 3837.950227] Lustre: Skipped 1 previous similar message [ 3843.045514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.048563] Lustre: Skipped 1 previous similar message [ 3843.802042] Lustre: server umount lustre-MDT0000 complete [ 3847.793862] LustreError: 112848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121270 with bad export cookie 13786273357460301160 [ 3847.799670] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3847.821217] LustreError: 112848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3848.297513] Lustre: server umount lustre-MDT0001 complete [ 3858.058401] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:34:40 (1787121280) [ 3861.872209] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3879.927880] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 3892.409828] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3929.884285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.112248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3930.131805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3930.195869] Lustre: lustre-MDT0000: new disk, initializing [ 3930.283931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3930.298132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3934.898556] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3945.989267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3946.220207] Lustre: 119729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3946.320882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3946.327594] Lustre: Skipped 1 previous similar message [ 3946.439894] Lustre: lustre-MDT0001: new disk, initializing [ 3946.564461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3946.604139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3946.624166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3952.425488] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3962.799531] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3969.318908] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3977.031838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3977.450436] Lustre: lustre-OST0000: new disk, initializing [ 3977.465845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3977.477966] Lustre: 121696:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3977.593691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3982.922078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3982.929075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3983.083861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3983.819637] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3993.588284] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3998.231481] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3998.432351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4003.171525] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4003.400852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4008.934027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4008.939235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.947382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4008.964555] Lustre: Skipped 1 previous similar message [ 4008.977579] Lustre: Skipped 1 previous similar message [ 4011.807796] Lustre: server umount lustre-OST0000 complete [ 4018.155721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4018.165104] LustreError: Skipped 1 previous similar message [ 4018.170140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.205783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4023.887610] Lustre: server umount lustre-MDT0000 complete [ 4024.289796] LustreError: 119735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4024.312502] LustreError: 119735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4029.309522] LustreError: 121695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121452 with bad export cookie 13786273357460302042 [ 4029.322735] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4029.331935] LustreError: 121695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4029.924534] Lustre: server umount lustre-MDT0001 complete [ 4039.969622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4040.386307] LustreError: 123913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4040.471553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4045.799080] LustreError: 123914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.342343] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4050.918151] LustreError: 123913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4057.059432] LustreError: 123914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4057.089904] LustreError: 123914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4057.151256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4061.686674] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4070.523169] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4078.109965] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4086.709765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4096.397938] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4108.202513] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4121.593923] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4126.995901] Lustre: server umount lustre-OST0000 complete [ 4132.336094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4132.346085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4132.353460] Lustre: Skipped 2 previous similar messages [ 4132.376693] Lustre: Skipped 3 previous similar messages [ 4141.587751] Lustre: server umount lustre-MDT0000 complete [ 4142.564572] LustreError: 123909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.587386] LustreError: 123909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4143.586647] Lustre: 108373:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787121550/real 1787121550] req@00000000b2e5fbeb x1873932161505920/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787121566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4143.638571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.673289] Lustre: 108375:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787121551/real 1787121551] req@00000000924f421c x1873932161506176/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787121567 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4145.892090] LustreError: 126814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121568 with bad export cookie 13786273357460302924 [ 4145.895287] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4145.902659] LustreError: 126814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4146.236801] Lustre: server umount lustre-MDT0001 complete [ 4160.555847] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4163.190831] Key type lgssc unregistered [ 4163.498592] LNet: 127569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4163.508955] LNetError: 127569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4163.526109] LNet: Removed LNI 192.168.202.103@tcp [ 4164.395421] Key type .llcrypt unregistered [ 4164.397078] Key type ._llcrypt unregistered [ 4184.485428] Key type ._llcrypt registered [ 4184.488705] Key type .llcrypt registered [ 4184.590989] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 4201.301342] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4202.435547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4202.535342] alg: No test for adler32 (adler32-zlib) [ 4203.586636] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 4203.871814] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4205.631210] Key type lgssc registered [ 4206.817645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4243.793608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4243.819744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4245.104459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4245.152518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4245.259515] Lustre: lustre-MDT0000: new disk, initializing [ 4245.312088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4245.325468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4249.960973] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4262.102905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4262.283699] Lustre: 130726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4262.356301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4262.363839] Lustre: Skipped 1 previous similar message [ 4262.490258] Lustre: lustre-MDT0001: new disk, initializing [ 4262.616204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4262.661737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4262.685819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4267.181819] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4278.839187] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4286.468475] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4293.828376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4294.108262] Lustre: lustre-OST0000: new disk, initializing [ 4294.111656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4294.119774] Lustre: 132692:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4294.192020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4300.421766] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4303.950579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4303.962344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4304.058859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4311.612567] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4316.758255] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4316.940267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4321.335575] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4321.508540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4324.841143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4324.841641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4324.869320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4324.877124] Lustre: Skipped 1 previous similar message [ 4324.907317] Lustre: Skipped 1 previous similar message [ 4329.931212] Lustre: server umount lustre-OST0000 complete [ 4335.075412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4335.083343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4335.092462] Lustre: Skipped 1 previous similar message [ 4335.102887] Lustre: Skipped 1 previous similar message [ 4340.192130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4340.207802] Lustre: Skipped 1 previous similar message [ 4341.114429] Lustre: server umount lustre-MDT0000 complete [ 4345.788768] LustreError: 130715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121768 with bad export cookie 17584248993476154947 [ 4345.810787] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4346.371601] Lustre: server umount lustre-MDT0001 complete [ 4358.011676] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:42:59 (1787121779) [ 4366.971214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4367.429416] LustreError: 135105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4367.453061] LustreError: 135105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4367.509613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4371.965543] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4372.966498] LustreError: 135106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.071211] LustreError: 135105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.177928] LustreError: 135106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.625446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4383.050600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4388.480956] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4397.084491] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4404.836751] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4407.776663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.786718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4407.792187] Lustre: Skipped 1 previous similar message [ 4413.477393] Lustre: server umount lustre-MDT0000 complete [ 4413.920874] LustreError: 136403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.946551] LustreError: 136403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4417.717323] LustreError: 135087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121840 with bad export cookie 17584248993476155829 [ 4417.724474] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4417.726109] LustreError: 135087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4418.022993] Lustre: server umount lustre-MDT0001 complete [ 4426.718924] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:44:08 (1787121848) [ 4432.954734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4448.485507] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4453.599650] LustreError: 137522:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 4479.396347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4487.312862] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4498.686613] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4501.021775] Lustre: server umount lustre-OST0000 complete [ 4511.255662] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:45:32 (1787121932) [ 4518.996556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4519.659228] LustreError: 138970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.784842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4525.776852] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4534.993967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4540.350684] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4549.571907] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4556.671071] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4563.957737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4564.380027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4564.387436] Lustre: Skipped 1 previous similar message [ 4571.155766] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4580.720735] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4591.083062] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4591.112808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4591.126447] Lustre: Skipped 1 previous similar message [ 4591.144759] Lustre: Skipped 5 previous similar messages [ 4595.686832] Lustre: server umount lustre-MDT0000 complete [ 4596.195058] LustreError: 138971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.231741] LustreError: 138971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4599.339544] LustreError: 138952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122022 with bad export cookie 17584248993476156494 [ 4599.353296] LustreError: 138952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4599.363519] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4599.723792] Lustre: server umount lustre-MDT0001 complete [ 4614.553294] Lustre: server umount lustre-OST0000 complete [ 4624.802964] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:47:26 (1787122046) [ 4633.215592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4633.681191] LustreError: 142631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.758250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4639.941880] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4651.139573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4656.725882] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4665.757901] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4672.373778] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4680.893089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4688.609856] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4698.164643] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4703.397949] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4703.943628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4704.869888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4710.690978] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4710.948477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4714.465829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4714.466039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4714.489548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4714.516615] Lustre: Skipped 2 previous similar messages [ 4714.558489] Lustre: Skipped 1 previous similar message [ 4719.451104] Lustre: server umount lustre-OST0000 complete [ 4724.712102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.737361] Lustre: Skipped 1 previous similar message [ 4730.446547] Lustre: server umount lustre-MDT0000 complete [ 4733.929237] LustreError: 142613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122156 with bad export cookie 17584248993476157565 [ 4733.940885] LustreError: 142613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4733.942161] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4734.208417] Lustre: server umount lustre-MDT0001 complete [ 4744.275546] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:49:26 (1787122166) [ 4750.924733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4766.495682] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4771.615485] LustreError: 146797:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 4797.344361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4797.360371] Lustre: Skipped 2 previous similar messages [ 4806.057472] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4815.637893] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4823.134713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4823.678522] LustreError: 146822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.704231] LustreError: 146822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4823.832155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4829.864407] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4839.184100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4844.034216] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4851.019262] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4857.653625] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4863.566264] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4864.043554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4870.582504] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4870.861674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4875.241847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4875.246996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4875.249755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4875.249766] Lustre: Skipped 4 previous similar messages [ 4875.255217] Lustre: Skipped 1 previous similar message [ 4879.434584] Lustre: server umount lustre-MDT0000 complete [ 4883.053437] LustreError: 146803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122306 with bad export cookie 17584248993476158587 [ 4883.063973] LustreError: 146803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4883.076638] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4883.477827] Lustre: server umount lustre-MDT0001 complete [ 4897.368689] Lustre: server umount lustre-OST0000 complete [ 4905.235964] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:52:07 (1787122327) [ 4911.161909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4926.751641] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4931.871497] LustreError: 150995:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 4965.381443] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4972.838638] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4979.998917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4980.474561] LustreError: 151020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4980.494369] LustreError: 151020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4980.609919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4984.916368] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4993.310325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.277791] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5006.033322] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5012.675190] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5021.641305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5021.769929] Lustre: lustre-OST0001: new disk, initializing [ 5021.773396] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5021.779646] Lustre: 153868:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5028.048977] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5029.901474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5029.910731] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5030.010725] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5037.312475] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5042.166980] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5042.394726] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5046.802535] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5047.048587] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5049.826287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5049.835198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5049.860177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5049.882049] Lustre: Skipped 2 previous similar messages [ 5063.135231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5063.238042] Lustre: server umount lustre-OST0000 complete [ 5070.824968] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5070.845676] LustreError: Skipped 1 previous similar message [ 5074.359654] Lustre: server umount lustre-OST0001 complete [ 5080.550978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5084.444884] Lustre: server umount lustre-MDT0000 complete [ 5088.515176] LustreError: 151001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122511 with bad export cookie 17584248993476159539 [ 5088.523019] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5088.528174] LustreError: 151001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5089.036761] Lustre: server umount lustre-MDT0001 complete [ 5118.584297] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:55:40 (1787122540) [ 5120.840921] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5122.635417] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5124.209836] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:55:46 (1787122546) [ 5131.493370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5131.718347] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5131.729402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5131.901402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5131.907479] Lustre: Skipped 6 previous similar messages [ 5136.221632] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5145.343590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5145.438575] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5145.467968] Lustre: 157427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5151.431138] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5159.543483] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5166.011212] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5174.711642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5175.112705] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5183.204147] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5185.224038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5192.916248] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5198.352867] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5198.532640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5204.185747] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5204.345354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5210.595275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5210.604829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5210.616974] Lustre: Skipped 5 previous similar messages [ 5210.622145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5210.635083] Lustre: Skipped 11 previous similar messages [ 5212.392484] Lustre: server umount lustre-OST0000 complete [ 5234.944047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5236.396258] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.413168] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5241.651965] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5249.051045] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5251.753086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.776783] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5255.886948] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5256.283991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5261.112978] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5261.381161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5267.398887] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92391067a000.ost_server_uuid 50 [ 5269.307625] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff92391067a000.ost_server_uuid in FULL state after 0 sec [ 5269.591247] LustreError: 159719:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.3@tcp) returned error from blocking AST (req@0000000043098738 x1873932846376064 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000076bac850/0xf407cd4b522ab18d lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.3@tcp remote: 0xd34a0a62cd486d6e expref: 5 pid: 157434 timeout: 5369 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5269.630692] LustreError: lustre-MDT0000: A client on nid 192.168.202.3@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5269.643158] LustreError: 157424:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.3@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000076bac850/0xf407cd4b522ab18d lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.3@tcp remote: 0xd34a0a62cd486d6e expref: 6 pid: 157434 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5275.119463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5275.126196] LustreError: Skipped 1 previous similar message [ 5280.676725] Lustre: server umount lustre-OST0000 complete [ 5291.542524] Lustre: server umount lustre-MDT0000 complete [ 5293.037929] LustreError: 157440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5293.072952] LustreError: 157440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5297.066843] LustreError: 157419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122720 with bad export cookie 17584248993476160722 [ 5297.089604] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5297.097390] LustreError: 157419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5303.550532] Lustre: server umount lustre-MDT0001 complete [ 5316.490660] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5320.094833] Key type lgssc unregistered [ 5320.375578] LNet: 163550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5320.388802] LNetError: 163550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5320.418168] LNet: Removed LNI 192.168.202.103@tcp [ 5321.224896] Key type .llcrypt unregistered [ 5321.227363] Key type ._llcrypt unregistered [ 5347.333642] Key type ._llcrypt registered [ 5347.336674] Key type .llcrypt registered [ 5347.435841] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5348.444828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5348.615629] alg: No test for adler32 (adler32-zlib) [ 5349.779313] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 5350.027842] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5351.735189] Key type lgssc registered [ 5352.957927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5361.618844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5361.638750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.060204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5368.541137] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5373.409374] LustreError: 164963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.416359] LustreError: 164963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5378.546836] LustreError: 164962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5378.644712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.754039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5386.514435] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5395.863388] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5403.290841] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5411.417549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5411.962984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5419.841395] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5429.777728] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5433.468355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5443.029626] Lustre: Failing over lustre-MDT0000 [ 5443.414507] Lustre: server umount lustre-MDT0000 complete [ 5444.066582] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.074047] LustreError: 164958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5444.078020] Lustre: Skipped 1 previous similar message [ 5444.109646] LustreError: 164958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5449.190272] LustreError: 164963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5449.210629] LustreError: 164963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5454.338817] LustreError: 164957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.369813] LustreError: 164957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5455.936446] LustreError: 164943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.3@tcp arrived at 1787122878 with bad export cookie 4706699443705900893 [ 5455.953881] LustreError: 164943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5460.452414] Lustre: 164272:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787122867/real 1787122867] req@00000000c3d478e2 x1873934048851968/t0(0) o400->MGC192.168.202.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787122883 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5460.469165] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5463.671145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5465.571039] LustreError: 164958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5465.600675] LustreError: 164958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5470.715717] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x41518e34f69e70a6 [ 5470.739229] Lustre: MGC192.168.202.103@tcp: Connection restored to 0@lo (at 0@lo) [ 5471.078995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5471.125380] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5475.814923] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5476.330866] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5476.334866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5478.156413] Lustre: lustre-MDT0000: Denying connection for new client 1358916b-e1f0-48b3-b8f2-392ce8f4436e (at 192.168.202.3@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5483.238533] Lustre: lustre-MDT0000: Denying connection for new client 1358916b-e1f0-48b3-b8f2-392ce8f4436e (at 192.168.202.3@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 5486.565363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5486.590906] Lustre: Skipped 1 previous similar message [ 5486.602574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5491.169609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5491.181935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5491.199236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5492.593346] Lustre: server umount lustre-OST0000 complete [ 5496.356168] LustreError: 168719:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5496.362911] Lustre: 168069:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5496.371366] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5496.409829] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5496.433055] LustreError: 168069:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000843e5f53 x1873934048872320/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5496.469097] LustreError: 168069:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5496.483204] LustreError: 168069:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5496.506608] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5496.801145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5496.810962] LustreError: 164957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5496.819682] LustreError: 164957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5496.884691] Lustre: server umount lustre-MDT0000 complete [ 5500.891628] LustreError: 164942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122923 with bad export cookie 4706699443705901222 [ 5500.894522] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5500.907045] LustreError: 164942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5503.333577] Lustre: server umount lustre-MDT0001 complete [ 5515.739341] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5518.597840] Key type lgssc unregistered [ 5519.029649] LNet: 169475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5519.037974] LNetError: 169475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5519.076908] LNet: Removed LNI 192.168.202.103@tcp [ 5520.068656] Key type .llcrypt unregistered [ 5520.070647] Key type ._llcrypt unregistered [ 5548.391188] Key type ._llcrypt registered [ 5548.398074] Key type .llcrypt registered [ 5548.541746] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5550.120591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5550.182423] alg: No test for adler32 (adler32-zlib) [ 5551.644562] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 5552.045310] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5553.807414] Key type lgssc registered [ 5555.209067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5564.244771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5564.256360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5565.782750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.053239] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5575.143759] LustreError: 170890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5575.166103] LustreError: 170890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5578.846667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5579.324792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5584.135971] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5591.915976] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5598.629474] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5606.835532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5607.243475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5610.374463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5615.746964] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5625.838033] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5635.043467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5635.048169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.067434] Lustre: Skipped 1 previous similar message [ 5635.067762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5640.169612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5640.179968] Lustre: Skipped 2 previous similar messages [ 5640.435297] Lustre: server umount lustre-OST0000 complete [ 5645.289770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5645.291482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5645.300495] Lustre: Skipped 1 previous similar message [ 5645.323810] Lustre: Skipped 1 previous similar message [ 5650.406295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5650.412140] Lustre: Skipped 1 previous similar message [ 5651.223198] Lustre: server umount lustre-MDT0000 complete [ 5655.520650] LustreError: 170884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5655.536938] LustreError: 170884:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5655.604509] LustreError: 170871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123078 with bad export cookie 2351215769607270243 [ 5655.615858] LustreError: 170871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5655.621797] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5655.935684] Lustre: server umount lustre-MDT0001 complete [ 5669.412652] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5674.081378] Key type lgssc unregistered [ 5674.333046] LNet: 174429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5674.340652] LNetError: 174429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5674.362988] LNet: Removed LNI 192.168.202.103@tcp [ 5675.288442] Key type .llcrypt unregistered [ 5675.291180] Key type ._llcrypt unregistered [ 5711.097844] Key type ._llcrypt registered [ 5711.105201] Key type .llcrypt registered [ 5711.217979] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5712.125876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5712.290659] alg: No test for adler32 (adler32-zlib) [ 5713.336624] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 5713.671383] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5715.327190] Key type lgssc registered [ 5716.302737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5723.336653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5723.352406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5724.630744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5728.399807] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5734.901655] LustreError: 176134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5734.923665] LustreError: 176134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5736.845171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5737.360025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5742.790092] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5754.202506] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5761.029338] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5769.283486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5769.769362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5776.222996] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5783.670750] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5794.361262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5802.167946] Lustre: server umount lustre-OST0000 complete [ 5809.120599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5809.128738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.153439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5809.635498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.666761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5814.756710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5814.766399] Lustre: Skipped 1 previous similar message [ 5816.937645] Lustre: server umount lustre-MDT0000 complete [ 5818.847155] Lustre: 175446:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787123225/real 1787123225] req@000000009db3f9fd x1873934429485696/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787123241 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5818.877810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5818.893328] LustreError: 177568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5818.924139] LustreError: 177568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5820.298392] LustreError: 176114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123243 with bad export cookie 5854000199241403405 [ 5820.303748] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5820.309640] LustreError: 176114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5820.578455] Lustre: server umount lustre-MDT0001 complete [ 5833.599693] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5837.268943] Key type lgssc unregistered [ 5837.602555] LNet: 179692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5837.610336] LNetError: 179692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5837.631225] LNet: Removed LNI 192.168.202.103@tcp [ 5838.645578] Key type .llcrypt unregistered [ 5838.647816] Key type ._llcrypt unregistered [ 5866.803603] Key type ._llcrypt registered [ 5866.806785] Key type .llcrypt registered [ 5866.904797] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5868.212492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5868.219624] alg: No test for adler32 (adler32-zlib) [ 5869.431657] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 5869.730037] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5871.383162] Key type lgssc registered [ 5872.414356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5883.331964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5883.361448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5884.782298] Lustre: *** cfs_fail_loc=135, val=0*** [ 5884.870038] LustreError: 181215:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5884.881350] LustreError: 181215:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.103@tcp: cfg command failed: rc = -2 [ 5884.892477] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5884.892477] [ 5884.899721] LustreError: MGC192.168.202.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5884.906650] LustreError: 181176:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5884.912607] LustreError: 181176:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5884.920920] LustreError: 181176:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5884.926961] LustreError: 181176:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5884.999476] Lustre: server umount lustre-MDT0000 complete [ 5885.004280] LustreError: 181176:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5892.597394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5893.081417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.997739] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5902.316991] LustreError: 181588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5902.347403] LustreError: 181588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5907.430969] LustreError: 181589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5908.863525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5909.598099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5915.695427] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5924.106763] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5931.159765] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5934.758118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5934.786987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5934.792519] Lustre: Skipped 1 previous similar message [ 5940.197881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5940.206246] Lustre: Skipped 1 previous similar message [ 5945.313839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5945.320724] Lustre: Skipped 1 previous similar message [ 5948.383166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5948.642735] Lustre: server umount lustre-MDT0000 complete [ 5950.434151] LustreError: 181589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5950.461963] LustreError: 181589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5952.579175] LustreError: 181570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123375 with bad export cookie 714754233084804582 [ 5952.596137] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5953.064297] Lustre: server umount lustre-MDT0001 complete [ 5965.024772] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5968.093712] Key type lgssc unregistered [ 5968.413542] LNet: 183914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5968.419305] LNetError: 183914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5968.439741] LNet: Removed LNI 192.168.202.103@tcp [ 5969.306898] Key type .llcrypt unregistered [ 5969.310439] Key type ._llcrypt unregistered [ 5994.200274] Key type ._llcrypt registered [ 5994.205785] Key type .llcrypt registered [ 5994.312132] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5994.611161] Key type .llcrypt unregistered [ 5994.612560] Key type ._llcrypt unregistered [ 6014.368140] Key type ._llcrypt registered [ 6014.370156] Key type .llcrypt registered [ 6014.545868] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6015.812355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6015.844631] alg: No test for adler32 (adler32-zlib) [ 6017.066529] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 6017.306049] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6018.999483] Key type lgssc registered [ 6020.602783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6029.776840] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6029.827723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6045.408370] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6050.527385] LustreError: 186052:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 6077.218529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6083.396046] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6091.292336] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6098.053476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6098.192892] LustreError: 185387:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bf180bf5 x1873934748223744/t0(0) o250->MGC192.168.202.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6098.708708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6098.777930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6103.310285] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6104.046920] LustreError: 186077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.061240] LustreError: 186077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6109.154419] LustreError: 186078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6111.206606] LustreError: 186077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6111.241817] LustreError: 186077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6114.342321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6114.765442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6119.560032] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6129.179763] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6138.556125] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6142.669930] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6160.354015] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6160.362092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.406727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6160.868869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.872779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.301216] Lustre: server umount lustre-OST0000 complete [ 6171.108215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6171.123667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.125186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6171.136177] Lustre: Skipped 1 previous similar message [ 6171.155811] Lustre: Skipped 2 previous similar messages [ 6173.662080] Lustre: server umount lustre-MDT0000 complete [ 6176.227524] LustreError: 187089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6176.249163] LustreError: 187089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6177.821471] LustreError: 186058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123600 with bad export cookie 9786596922951163814 [ 6177.822314] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6177.829253] LustreError: 186058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6178.233333] Lustre: server umount lustre-MDT0001 complete [ 6189.991175] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6192.836276] Key type lgssc unregistered [ 6193.159208] LNet: 190022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6193.184977] LNetError: 190022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6193.202715] LNet: Removed LNI 192.168.202.103@tcp [ 6193.993146] Key type .llcrypt unregistered [ 6193.995507] Key type ._llcrypt unregistered [ 6220.716216] Key type ._llcrypt registered [ 6220.718823] Key type .llcrypt registered [ 6220.845761] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6222.140727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6222.176982] alg: No test for adler32 (adler32-zlib) [ 6223.320707] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 6223.620555] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6225.359175] Key type lgssc registered [ 6226.253804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6233.997104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6234.023257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.544500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.167353] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6245.869137] LustreError: 191436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6245.882777] LustreError: 191436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6249.242765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.685331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6255.229444] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6263.941555] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6271.007069] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6279.850087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6280.368726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.418562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6286.822869] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6295.343705] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6300.808623] Lustre: Failing over lustre-MDT0000 [ 6301.209399] Lustre: server umount lustre-MDT0000 complete [ 6302.459720] LustreError: 192642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.250732] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6305.258569] LustreError: 191436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.278525] Lustre: Skipped 1 previous similar message [ 6305.293223] LustreError: 191436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6307.575505] LustreError: 192642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.684627] LustreError: 192642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.714637] LustreError: 192642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6321.647268] Lustre: 190748:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787123728/real 1787123728] req@0000000099ac7b32 x1873934964256640/t0(0) o400->MGC192.168.202.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787123744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6321.675458] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6321.713299] LustreError: 191436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6321.729880] LustreError: 191436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6321.800225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.880839] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa821eb85d05d0869 [ 6331.892365] Lustre: MGC192.168.202.103@tcp: Connection restored to 0@lo (at 0@lo) [ 6332.129536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6332.201633] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6334.214238] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6337.527884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6337.560965] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6337.631910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6337.766416] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6343.048499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6349.270590] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6360.036660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6360.055190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.084846] Lustre: Skipped 1 previous similar message [ 6360.098908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6361.379960] Lustre: server umount lustre-OST0000 complete [ 6368.225361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6368.231160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6368.240022] Lustre: Skipped 1 previous similar message [ 6368.244385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6368.248863] Lustre: Skipped 1 previous similar message [ 6371.658652] Lustre: server umount lustre-MDT0000 complete [ 6373.348316] LustreError: 191431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6373.358261] LustreError: 191431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6375.748401] LustreError: 195221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123798 with bad export cookie 12115223432562280553 [ 6375.752189] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6375.768199] LustreError: 195221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6385.417931] Lustre: server umount lustre-MDT0001 complete [ 6398.307204] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6401.305819] Key type lgssc unregistered [ 6401.594703] LNet: 196284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6401.602760] LNetError: 196284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6402.670208] LNet: Removed LNI 192.168.202.103@tcp [ 6403.913417] Key type .llcrypt unregistered [ 6403.915720] Key type ._llcrypt unregistered [ 6434.093695] Key type ._llcrypt registered [ 6434.097714] Key type .llcrypt registered [ 6434.216961] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6435.678238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6435.711766] alg: No test for adler32 (adler32-zlib) [ 6436.879567] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 6437.154611] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6438.847255] Key type lgssc registered [ 6440.212464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6448.593878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6448.618073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6449.909209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6454.693072] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6459.364798] LustreError: 197695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6459.375851] LustreError: 197695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6463.416705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.936772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6468.377337] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6475.670927] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6481.749901] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6489.226744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6489.545966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6494.729312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6495.501923] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6504.730389] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6507.778068] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6514.985562] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6530.722311] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6535.649084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6535.667685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.693524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6539.748136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6539.752589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6539.753843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6539.778561] Lustre: Skipped 1 previous similar message [ 6541.144758] Lustre: server umount lustre-OST0000 complete [ 6544.864402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6544.866368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6544.873547] Lustre: Skipped 1 previous similar message [ 6544.881322] Lustre: Skipped 1 previous similar message [ 6549.984441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6549.987185] Lustre: Skipped 1 previous similar message [ 6555.106100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6555.114467] Lustre: Skipped 1 previous similar message [ 6559.204640] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6559.424306] Lustre: server umount lustre-MDT0000 complete [ 6563.424760] LustreError: 197676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123986 with bad export cookie 11797272497437913573 [ 6563.425437] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6563.433227] LustreError: 197676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6563.717122] Lustre: server umount lustre-MDT0001 complete [ 6577.506553] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6581.679618] Key type lgssc unregistered [ 6582.092063] LNet: 201545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6582.108365] LNetError: 201545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6583.161727] LNet: Removed LNI 192.168.202.103@tcp [ 6584.522042] Key type .llcrypt unregistered [ 6584.527718] Key type ._llcrypt unregistered [ 6605.834490] Key type ._llcrypt registered [ 6605.844227] Key type .llcrypt registered [ 6605.951362] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6607.051286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6607.084735] alg: No test for adler32 (adler32-zlib) [ 6608.122743] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 6608.373615] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6610.055184] Key type lgssc registered [ 6611.329780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6619.021817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6619.035151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.533611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6625.324384] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6630.883399] LustreError: 202821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6630.925259] LustreError: 202821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6636.004612] LustreError: 202822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6636.515728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6637.080755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6642.884763] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6652.085930] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6658.326226] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6666.187367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6666.573989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6672.795377] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6680.997238] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6690.157641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6696.692242] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6710.092142] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6719.667810] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6725.393156] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6736.390742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.391967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6736.413235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6736.420788] Lustre: Skipped 1 previous similar message [ 6741.082520] Lustre: server umount lustre-OST0000 complete [ 6746.594665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.620624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6746.624878] Lustre: Skipped 2 previous similar messages [ 6746.654571] Lustre: Skipped 1 previous similar message [ 6751.712267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6751.720167] Lustre: Skipped 1 previous similar message [ 6756.834145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6756.849201] Lustre: Skipped 1 previous similar message [ 6759.391387] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6760.004953] Lustre: server umount lustre-MDT0000 complete [ 6761.952432] LustreError: 202816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6761.985815] LustreError: 202816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6764.096714] LustreError: 202802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124187 with bad export cookie 16361107610415018643 [ 6764.104967] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6764.433601] Lustre: server umount lustre-MDT0001 complete [ 6776.754126] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6780.098569] Key type lgssc unregistered [ 6780.528811] LNet: 207774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6780.534048] LNetError: 207774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6780.544781] LNet: Removed LNI 192.168.202.103@tcp [ 6781.667416] Key type .llcrypt unregistered [ 6781.669328] Key type ._llcrypt unregistered [ 6812.029319] Key type ._llcrypt registered [ 6812.038887] Key type .llcrypt registered [ 6812.230748] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6814.772930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6814.792992] alg: No test for adler32 (adler32-zlib) [ 6815.849663] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 6816.202399] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6817.943120] Key type lgssc registered [ 6819.185325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6828.568694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6828.609341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6830.242027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6835.502344] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6840.806641] LustreError: 209191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.844321] LustreError: 209191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6845.923470] LustreError: 209192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6846.992210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6847.572098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6854.214418] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6863.941864] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6872.298465] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6880.093147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6880.543571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6887.079923] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6895.648742] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6901.061387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6918.626081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6918.633876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6918.649732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6921.696161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6921.697775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6921.715404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6924.387594] Lustre: server umount lustre-OST0000 complete [ 6929.378265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6929.398120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6929.421409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6929.429353] Lustre: Skipped 1 previous similar message [ 6931.938539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6931.939665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6934.776923] Lustre: server umount lustre-MDT0000 complete [ 6937.057729] LustreError: 209192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6937.081335] LustreError: 209192:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6938.881824] LustreError: 209171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124361 with bad export cookie 1606323216870304779 [ 6938.896555] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6938.910268] LustreError: 209171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6939.598453] Lustre: server umount lustre-MDT0001 complete [ 6952.932177] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6957.119742] Key type lgssc unregistered [ 6957.513326] LNet: 212902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6957.527025] LNetError: 212902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6957.549029] LNet: Removed LNI 192.168.202.103@tcp [ 6958.685902] Key type .llcrypt unregistered [ 6958.687868] Key type ._llcrypt unregistered [ 6980.285756] Key type ._llcrypt registered [ 6980.288729] Key type .llcrypt registered [ 6980.494794] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6982.835361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6982.981637] alg: No test for adler32 (adler32-zlib) [ 6984.189615] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 6984.595323] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6986.351187] Key type lgssc registered [ 6988.459715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6997.341845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6997.397296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6999.038762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7004.123555] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7008.225483] LustreError: 214169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.235935] LustreError: 214169:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7013.345980] LustreError: 214170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7014.699489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7015.359455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7020.874176] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7029.442058] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7035.764492] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7043.645854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7044.020657] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7044.174464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7049.215695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7051.456634] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7060.176070] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7079.908967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7079.921603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7079.960153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7080.935838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7080.938553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7080.987524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7080.996194] Lustre: Skipped 1 previous similar message [ 7083.166804] Lustre: server umount lustre-OST0000 complete [ 7091.171924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7091.181879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7091.192175] Lustre: Skipped 1 previous similar message [ 7091.215971] Lustre: Skipped 1 previous similar message [ 7093.716282] Lustre: server umount lustre-MDT0000 complete [ 7096.294217] LustreError: 214169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7096.321783] LustreError: 214169:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7098.064655] LustreError: 214149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124521 with bad export cookie 8199756817064423698 [ 7098.075377] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7098.083314] LustreError: 214149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7098.753896] Lustre: server umount lustre-MDT0001 complete [ 7111.981859] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7115.331810] Key type lgssc unregistered [ 7115.630696] LNet: 217855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7115.638823] LNetError: 217855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7115.656898] LNet: Removed LNI 192.168.202.103@tcp [ 7116.563117] Key type .llcrypt unregistered [ 7116.565902] Key type ._llcrypt unregistered [ 7137.571354] Key type ._llcrypt registered [ 7137.574153] Key type .llcrypt registered [ 7137.695531] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7138.839292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7139.099207] alg: No test for adler32 (adler32-zlib) [ 7140.192600] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 7140.441480] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7142.119167] Key type lgssc registered [ 7143.514322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7152.571506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7152.602773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7154.159351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7158.981697] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7164.387253] LustreError: 219126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7164.416732] LustreError: 219126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7168.645287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7169.014602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7173.924701] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7183.564738] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7191.409777] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7199.948310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7200.266894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7207.468547] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7216.737505] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7224.884610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7240.385548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7240.498732] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7241.014618] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7248.879830] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7259.124153] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7260.843582] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7273.347892] Lustre: Permanently deactivating lustre-OST0001 [ 7273.359507] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7281.290790] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7287.812288] Lustre: Permanently reactivating lustre-OST0001 [ 7287.835335] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7287.840880] Lustre: Skipped 2 previous similar messages [ 7288.702917] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7288.722084] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7288.737900] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7288.749257] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7288.788542] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7304.161591] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7304.167770] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7304.174224] Lustre: Skipped 1 previous similar message [ 7304.178480] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7304.184932] Lustre: Skipped 1 previous similar message [ 7306.869982] Lustre: server umount lustre-OST0001 complete [ 7309.811272] LustreError: 221620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7309.839328] LustreError: 221620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7311.840194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7311.846865] LustreError: Skipped 1 previous similar message [ 7311.851675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.864917] Lustre: Skipped 1 previous similar message [ 7311.870787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7314.916250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7314.916525] LustreError: 221620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7314.936520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7315.003131] LustreError: 221620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7318.006602] Lustre: server umount lustre-OST0000 complete [ 7322.594447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7322.613122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7322.640138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7322.646620] Lustre: Skipped 1 previous similar message [ 7328.678379] Lustre: server umount lustre-MDT0000 complete [ 7330.274487] LustreError: 219126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7330.296651] LustreError: 219126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7333.273147] LustreError: 219861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124756 with bad export cookie 10749718721289866731 [ 7333.294769] LustreError: 219861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7333.298223] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7333.662990] Lustre: server umount lustre-MDT0001 complete [ 7347.156223] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7350.474301] Key type lgssc unregistered [ 7350.854392] LNet: 224221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7350.859129] LNetError: 224221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7350.888413] LNet: Removed LNI 192.168.202.103@tcp [ 7352.190878] Key type .llcrypt unregistered [ 7352.194438] Key type ._llcrypt unregistered [ 7400.358727] Key type ._llcrypt registered [ 7400.367571] Key type .llcrypt registered [ 7400.505718] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7402.490092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7402.520835] alg: No test for adler32 (adler32-zlib) [ 7403.809534] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 7404.152106] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7405.927396] Key type lgssc registered [ 7407.996300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7420.555400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7420.587579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7422.010759] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7422.050146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7422.180324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7428.163768] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7440.627889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.696535] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7440.718953] Lustre: 226429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7440.935657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7445.008537] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7455.689529] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7464.108123] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7474.357316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7474.573447] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7474.925926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.208034] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7484.262774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7492.961431] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7497.329524] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7501.613271] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7512.100568] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7523.926954] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7532.094756] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7550.882181] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7550.892293] Lustre: Skipped 1 previous similar message [ 7586.020327] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7586.032125] Lustre: Skipped 3 previous similar messages [ 7650.120700] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7650.143390] Lustre: Skipped 7 previous similar messages [ 7684.070408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7684.079870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7684.091928] LustreError: Skipped 1 previous similar message [ 7684.101910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7684.113959] Lustre: Skipped 1 previous similar message [ 7686.104900] Lustre: server umount lustre-OST0000 complete [ 7691.745510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7691.749704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7691.761706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7691.774985] Lustre: Skipped 1 previous similar message [ 7694.320018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7694.327859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7696.680262] Lustre: server umount lustre-MDT0000 complete [ 7700.984105] LustreError: 226421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125123 with bad export cookie 15834632290890547575 [ 7700.988247] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7700.997177] LustreError: 226421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7701.472296] Lustre: server umount lustre-MDT0001 complete [ 7715.177089] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7718.068761] Key type lgssc unregistered [ 7718.335809] LNet: 231229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7718.350634] LNetError: 231229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7718.388197] LNet: Removed LNI 192.168.202.103@tcp [ 7719.396835] Key type .llcrypt unregistered [ 7719.399115] Key type ._llcrypt unregistered [ 7748.334966] Key type ._llcrypt registered [ 7748.337414] Key type .llcrypt registered [ 7748.434775] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7749.345463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7749.386728] alg: No test for adler32 (adler32-zlib) [ 7750.376917] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 7750.586198] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7752.311269] Key type lgssc registered [ 7753.469290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7761.065332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7761.095896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7762.686912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.298744] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7768.044677] LustreError: 232643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.066689] LustreError: 232643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7772.133713] LustreError: 232642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7776.730445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.376131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7782.933312] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7793.222473] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7800.957853] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7808.575634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7808.980371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7817.233546] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7828.745812] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7832.573888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7847.913154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7847.918675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7847.920769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7847.924401] LustreError: Skipped 1 previous similar message [ 7853.035714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7853.052739] Lustre: Skipped 1 previous similar message [ 7853.623692] Lustre: server umount lustre-OST0000 complete [ 7858.148507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.152364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7858.165115] Lustre: Skipped 2 previous similar messages [ 7858.184350] Lustre: Skipped 2 previous similar messages [ 7863.267125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7863.271502] Lustre: Skipped 1 previous similar message [ 7864.209119] Lustre: server umount lustre-MDT0000 complete [ 7868.177752] LustreError: 234487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125291 with bad export cookie 11089600250594138929 [ 7868.186672] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7868.189295] LustreError: 234487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7868.690113] Lustre: server umount lustre-MDT0001 complete [ 7882.367261] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7885.089285] Key type lgssc unregistered [ 7885.369563] LNet: 236337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7885.381221] LNetError: 236337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7885.392796] LNet: Removed LNI 192.168.202.103@tcp [ 7886.153282] Key type .llcrypt unregistered [ 7886.158529] Key type ._llcrypt unregistered [ 7913.941367] Key type ._llcrypt registered [ 7913.943582] Key type .llcrypt registered [ 7914.080800] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7915.549674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7915.680309] alg: No test for adler32 (adler32-zlib) [ 7916.721984] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 7917.001987] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7918.679170] Key type lgssc registered [ 7920.056253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8192.768715] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 8196.247866] Key type lgssc unregistered [ 8196.611807] LNet: 238382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8196.625475] LNetError: 238382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8196.648378] LNet: Removed LNI 192.168.202.103@tcp [ 8197.809319] Key type .llcrypt unregistered [ 8197.815854] Key type ._llcrypt unregistered [ 8231.018247] Key type ._llcrypt registered [ 8231.024546] Key type .llcrypt registered [ 8231.182910] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 8251.977883] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8253.024344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8253.110404] alg: No test for adler32 (adler32-zlib) [ 8254.213788] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 8254.558093] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 8256.215834] Key type lgssc registered [ 8257.367815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8301.020271] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7985 sec ========== 03:48:42 (1787125722) [ 8303.548587] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:48:45 (1787125725) === [ 8307.535797] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:48:49 (1787125729) === [ 8359.405561] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 8363.444486] Key type lgssc unregistered [ 8364.000372] LNet: 243630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8364.014276] LNetError: 243630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8364.043207] LNet: Removed LNI 192.168.202.103@tcp [ 8365.353259] Key type .llcrypt unregistered [ 8365.360112] Key type ._llcrypt unregistered