[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 624065620 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002329] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006840] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010164] LSM: Security Framework initializing [ 0.011080] Yama: becoming mindful. [ 0.012059] SELinux: Initializing. [ 0.013075] *** VALIDATE selinux *** [ 0.023019] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030000] *** VALIDATE tmpfs *** [ 0.030000] *** VALIDATE proc *** [ 0.030000] *** VALIDATE cgroup *** [ 0.030000] *** VALIDATE cgroup2 *** [ 0.030000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.030000] Spectre V2 : User space: Vulnerable [ 0.030000] Speculative Store Bypass: Vulnerable [ 0.030000] debug: unmapping init [mem 0xffffffff96c59000-0xffffffff96c60fff] [ 0.030000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.030767] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.031020] ... version: 2 [ 0.032010] ... bit width: 48 [ 0.033008] ... generic registers: 4 [ 0.034010] ... value mask: 0000ffffffffffff [ 0.035011] ... max period: 00007fffffffffff [ 0.036013] ... fixed-purpose events: 3 [ 0.037011] ... event mask: 000000070000000f [ 0.038311] rcu: Hierarchical SRCU implementation. [ 0.040591] smp: Bringing up secondary CPUs ... [ 0.041601] x86: Booting SMP configuration: [ 0.042026] .... node #0, CPUs: #1 #2 #3 [ 0.049263] smp: Brought up 1 node, 4 CPUs [ 0.051018] smpboot: Max logical packages: 1 [ 0.052011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.111584] node 0 deferred pages initialised in 57ms [ 0.116173] devtmpfs: initialized [ 0.117410] x86/mm: Memory block size: 128MB [ 0.120571] gcov: version magic: 0x41383552 [ 0.122456] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.123072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.124314] pinctrl core: initialized pinctrl subsystem [ 0.125398] [ 0.125791] ************************************************************* [ 0.126014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.127012] ** ** [ 0.128014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.129028] ** ** [ 0.130026] ** This means that this kernel is built to expose internal ** [ 0.131020] ** IOMMU data structures, which may compromise security on ** [ 0.132028] ** your system. ** [ 0.133017] ** ** [ 0.134017] ** If you see this message and you are not debugging the ** [ 0.135015] ** kernel, report this immediately to your vendor! ** [ 0.136014] ** ** [ 0.137014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138015] ************************************************************* [ 0.149545] NET: Registered protocol family 16 [ 0.150502] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.151089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.152088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.159020] cpuidle: using governor menu [ 0.162435] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.164450] PCI: Using configuration type 1 for base access [ 0.166159] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.172355] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.173028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.175307] cryptd: max_cpu_qlen set to 1000 [ 0.176861] ACPI: Added _OSI(Module Device) [ 0.177000] ACPI: Added _OSI(Processor Device) [ 0.177000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.179011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.183785] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.190104] ACPI: Interpreter enabled [ 0.191045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.192009] ACPI: Using IOAPIC for interrupt routing [ 0.193088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.196387] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.208624] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.210030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.212017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.214078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.218536] acpiphp: Slot [2] registered [ 0.220093] acpiphp: Slot [5] registered [ 0.221084] acpiphp: Slot [6] registered [ 0.222105] acpiphp: Slot [7] registered [ 0.224086] acpiphp: Slot [8] registered [ 0.225091] acpiphp: Slot [9] registered [ 0.226083] acpiphp: Slot [10] registered [ 0.227078] acpiphp: Slot [3] registered [ 0.228071] acpiphp: Slot [4] registered [ 0.229095] acpiphp: Slot [11] registered [ 0.230193] acpiphp: Slot [12] registered [ 0.232078] acpiphp: Slot [13] registered [ 0.233114] acpiphp: Slot [14] registered [ 0.234073] acpiphp: Slot [15] registered [ 0.235089] acpiphp: Slot [16] registered [ 0.237078] acpiphp: Slot [17] registered [ 0.238088] acpiphp: Slot [18] registered [ 0.239066] acpiphp: Slot [19] registered [ 0.240069] acpiphp: Slot [20] registered [ 0.241000] acpiphp: Slot [21] registered [ 0.241000] acpiphp: Slot [22] registered [ 0.241091] acpiphp: Slot [23] registered [ 0.242000] acpiphp: Slot [24] registered [ 0.244065] acpiphp: Slot [25] registered [ 0.246086] acpiphp: Slot [26] registered [ 0.247079] acpiphp: Slot [27] registered [ 0.248092] acpiphp: Slot [28] registered [ 0.250090] acpiphp: Slot [29] registered [ 0.253142] acpiphp: Slot [30] registered [ 0.254198] acpiphp: Slot [31] registered [ 0.255069] PCI host bridge to bus 0000:00 [ 0.256013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.258015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.260015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.262017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.264022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.266018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.268154] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.269925] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.273248] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.282015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.286657] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.289012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.290017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.292013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.294660] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.295000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.297037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.300878] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.306011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.320015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.325015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.332047] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.341017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.367031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.389027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.450235] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.463027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.608020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.619157] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.626017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.632016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.661028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.688526] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.697000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.697000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.733027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.771854] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.791024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.805024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.849030] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.042407] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.047024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.053020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.069025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.083307] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.089120] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.093456] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.096000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.099357] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.108515] iommu: Default domain type: Passthrough [ 1.115151] SCSI subsystem initialized [ 1.126262] ACPI: bus type USB registered [ 1.200178] usbcore: registered new interface driver usbfs [ 1.206171] usbcore: registered new interface driver hub [ 1.210426] usbcore: registered new device driver usb [ 1.216223] pps_core: LinuxPPS API ver. 1 registered [ 1.219016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.222082] PTP clock support registered [ 1.225061] EDAC MC: Ver: 3.0.0 [ 1.227248] PCI: Using ACPI for IRQ routing [ 1.230109] NetLabel: Initializing [ 1.231010] NetLabel: domain hash size = 128 [ 1.233017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.235119] NetLabel: unlabeled traffic allowed by default [ 1.238234] vgaarb: loaded [ 1.239432] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.240012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.297367] clocksource: Switched to clocksource kvm-clock [ 1.917180] VFS: Disk quotas dquot_6.6.0 [ 1.922519] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.935199] *** VALIDATE ramfs *** [ 1.942558] *** VALIDATE hugetlbfs *** [ 1.953383] pnp: PnP ACPI init [ 1.965952] pnp: PnP ACPI: found 6 devices [ 2.073066] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 2.092758] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 2.104611] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 2.114503] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 2.120364] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 2.131934] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 2.145214] NET: Registered protocol family 2 [ 2.159201] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 2.180893] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 2.191677] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 2.216958] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 2.240888] TCP: Hash tables configured (established 65536 bind 65536) [ 2.257128] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 2.270868] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.294829] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.306243] NET: Registered protocol family 1 [ 2.315303] RPC: Registered named UNIX socket transport module. [ 2.326209] RPC: Registered udp transport module. [ 2.335684] RPC: Registered tcp transport module. [ 2.341047] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.344182] NET: Registered protocol family 44 [ 2.346236] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 2.348230] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 2.368144] pci 0000:00:00.0: quirk_passive_release+0x0/0x90 took 19470 usecs [ 2.385263] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.393902] PCI: CLS 0 bytes, default 64 [ 2.400878] Unpacking initramfs... [ 6.475789] debug: unmapping init [mem 0xffff8be27cc54000-0xffff8be27ffbffff] [ 6.481403] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 6.483787] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 6.487135] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 9.377391] Initialise system trusted keyrings [ 9.388133] Key type blacklist registered [ 9.399553] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 9.451061] zbud: loaded [ 9.469508] *** VALIDATE nfs *** [ 9.475148] *** VALIDATE nfs4 *** [ 9.484708] pstore: using deflate compression [ 9.652807] Platform Keyring initialized [ 10.387405] NET: Registered protocol family 38 [ 10.397222] Key type asymmetric registered [ 10.399077] Asymmetric key parser 'x509' registered [ 10.416134] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 10.430692] io scheduler mq-deadline registered [ 10.431839] io scheduler kyber registered [ 10.433469] io scheduler bfq registered [ 10.470569] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 10.473572] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 10.486613] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 10.492226] ACPI: Power Button [PWRF] [ 10.506580] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 10.535726] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 10.586980] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 10.619455] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 10.694556] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 10.775904] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 10.872725] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 10.904368] Non-volatile memory driver v1.3 [ 10.913181] Linux agpgart interface v0.103 [ 11.038964] virtio_blk virtio1: [vda] 134776 512-byte logical blocks (69.0 MB/65.8 MiB) [ 11.063138] vda: detected capacity change from 0 to 69005312 [ 11.119038] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 11.126744] vdb: detected capacity change from 0 to 1073741824 [ 11.171762] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 11.178210] vdc: detected capacity change from 0 to 2621440000 [ 11.241061] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 11.257578] vdd: detected capacity change from 0 to 2621440000 [ 11.326612] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 11.333885] vde: detected capacity change from 0 to 4294967296 [ 11.393620] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 11.404793] vdf: detected capacity change from 0 to 4294967296 [ 11.428535] libphy: Fixed MDIO Bus: probed [ 11.460702] usbcore: registered new interface driver usbserial_generic [ 11.471144] usbserial: USB Serial support registered for generic [ 11.475676] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 11.488784] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 11.498197] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 11.508344] mousedev: PS/2 mouse device common for all mice [ 11.517551] rtc_cmos 00:05: RTC can wake from S4 [ 11.531535] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 11.536383] rtc_cmos 00:05: registered as rtc0 [ 11.538629] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 11.540847] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 11.557226] intel_pstate: CPU model not supported [ 11.558743] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 11.576112] hid: raw HID events driver (C) Jiri Kosina [ 11.578436] usbcore: registered new interface driver usbhid [ 11.580630] usbhid: USB HID core driver [ 11.581922] drop_monitor: Initializing network drop monitor service [ 11.583971] Initializing XFRM netlink socket [ 11.588862] NET: Registered protocol family 10 [ 11.604971] Segment Routing with IPv6 [ 11.606097] NET: Registered protocol family 17 [ 11.608199] mpls_gso: MPLS GSO support [ 11.624978] RAS: Correctable Errors collector initialized. [ 11.631357] AVX version of gcm_enc/dec engaged. [ 11.636241] AES CTR mode by8 optimization enabled [ 12.045532] sched_clock: Marking stable (12045332677, 0)->(13656270249, -1610937572) [ 12.086808] registered taskstats version 1 [ 12.113283] Loading compiled-in X.509 certificates [ 12.120809] zswap: loaded using pool lzo/zbud [ 12.246368] Key type big_key registered [ 12.352928] Key type encrypted registered [ 12.359740] ima: No TPM chip found, activating TPM-bypass! [ 12.365806] ima: Allocated hash algorithm: sha1 [ 12.373627] ima: No architecture policies found [ 12.377292] evm: Initialising EVM extended attributes: [ 12.389796] evm: security.selinux [ 12.391703] evm: security.ima [ 12.396948] evm: security.capability [ 12.402212] evm: HMAC attrs: 0x1 [ 12.434298] rtc_cmos 00:05: setting system clock to 2026-05-25 07:08:47 UTC (1779692927) [ 12.461265] debug: unmapping init [mem 0xffffffff97c03000-0xffffffff97dfffff] [ 12.466358] debug: unmapping init [mem 0xffffffff96982000-0xffffffff96c58fff] [ 12.481093] Write protecting the kernel read-only data: 28672k [ 12.490374] debug: unmapping init [mem 0xffffffff95003000-0xffffffff951fffff] [ 12.494371] debug: unmapping init [mem 0xffffffff95914000-0xffffffff959fffff] [ 12.623143] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.641102] systemd[1]: Detected virtualization kvm. [ 12.646333] systemd[1]: Detected architecture x86-64. [ 12.649140] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 12.817731] systemd[1]: No hostname configured. [ 12.825077] systemd[1]: Set hostname to . [ 12.830523] random: systemd: uninitialized urandom read (16 bytes read) [ 12.834129] systemd[1]: Initializing machine ID from random generator. [ 13.007120] random: ln: uninitialized urandom read (6 bytes read) [ 13.307936] random: systemd: uninitialized urandom read (16 bytes read) [ 13.316859] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 13.332780] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 13.340247] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 15.935487] device-mapper: uevent: version 1.0.3 [ 15.945568] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 18.724913] random: fast init done [ 18.749364] scsi host0: ata_piix [ 18.774398] scsi host1: ata_piix [ 18.785488] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 18.792588] virtio_net virtio0 ens2: renamed from eth0 [ 18.798260] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 23.959281] random: crng init done [ 23.962117] random: 7 urandom warning(s) missed due to ratelimiting [ 28.202993] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 30.031881] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 35.531801] printk: systemd: 26 output lines suppressed due to ratelimiting [ 36.912271] SELinux: Disabled at runtime. [ 37.080087] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 37.111138] systemd[1]: Detected virtualization kvm. [ 37.114367] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 39.734192] systemd[1]: initrd-switch-root.service: Succeeded. [ 39.747502] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 39.767972] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 39.781377] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 39.793549] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 39.846590] systemd[1]: Starting Journal Service... Starting Journal Service... [ 39.877236] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ 40.127157] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 42.127441] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 43.974748] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 44.224824] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 45.273619] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 45.483655] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 52.340553] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (13s / no limit)[ 53.088283] NFS: Registering the id_resolver key type [ 53.091547] Key type id_resolver registered [ 53.094362] Key type id_legacy registered [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ 56.392308] hrtimer: interrupt took 2465903 ns [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg415-server login: [ 125.739989] libcfs: loading out-of-tree module taints kernel. [ 125.802383] Key type ._llcrypt registered [ 125.805733] Key type .llcrypt registered [ 125.937906] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 144.161837] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 145.633215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 145.641676] alg: No test for adler32 (adler32-zlib) [ 146.971974] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 147.769172] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 149.568334] Key type lgssc registered [ 151.338066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 170.601768] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 214.099469] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 227.583718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 227.608066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.970730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 229.047274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 229.160607] Lustre: lustre-MDT0000: new disk, initializing [ 229.290885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.311520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 233.945528] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.772399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 247.894724] Lustre: 6512:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 247.937862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 247.941621] Lustre: Skipped 1 previous similar message [ 248.048969] Lustre: lustre-MDT0001: new disk, initializing [ 248.157903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 248.196391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 248.220764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 251.848281] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.446734] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 265.841880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 266.104832] Lustre: lustre-OST0000: new disk, initializing [ 266.108525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 266.114112] Lustre: 8417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 266.171515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 271.415690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 271.424966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 271.550436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 272.621368] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 289.037969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 289.175238] Lustre: lustre-OST0001: new disk, initializing [ 289.177881] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 289.182816] Lustre: 9472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 289.246211] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 296.163491] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 297.085191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 297.095630] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 297.143805] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 309.149872] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 316.677198] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 323.542410] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing check_logdir /tmp/testlogs/ [ 328.502205] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing yml_node [ 332.915910] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 335.282211] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 338.003749] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 339.711856] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon May 25 03:14:13 EDT 2026 [ 358.082476] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 360.060526] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 376.288784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 376.306498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.326274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.363311] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.365076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.374963] Lustre: Skipped 1 previous similar message [ 379.381867] Lustre: Skipped 1 previous similar message [ 382.180882] Lustre: server umount lustre-MDT0000 complete [ 386.029849] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693301 with bad export cookie 12655028913543303644 [ 386.032393] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 386.038308] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 389.600853] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.603431] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 389.603897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 389.603902] Lustre: Skipped 1 previous similar message [ 389.623019] Lustre: Skipped 2 previous similar messages [ 389.660304] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 391.649239] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 391.650049] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 391.675097] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 391.696764] Lustre: Skipped 2 previous similar messages [ 396.769041] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.774012] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 396.784095] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 396.815312] Lustre: Skipped 1 previous similar message [ 400.361425] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 400.790112] Lustre: server umount lustre-MDT0001 complete [ 405.603531] Lustre: server umount lustre-OST0000 complete [ 409.764762] Lustre: server umount lustre-OST0001 complete [ 416.612480] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 427.602298] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 465.880641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.240375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 466.300928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 466.419571] Lustre: lustre-MDT0000: new disk, initializing [ 466.502668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.521218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 471.391780] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 481.806396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.908973] Lustre: 14990:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 481.951791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 481.955272] Lustre: Skipped 1 previous similar message [ 482.030755] Lustre: lustre-MDT0001: new disk, initializing [ 482.125573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 482.170451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 482.184690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 486.579924] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 494.926340] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 498.993480] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 506.024518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 506.179608] Lustre: lustre-OST0000: new disk, initializing [ 506.182617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 506.187089] Lustre: 16731:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 506.243356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.203376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 508.224786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 508.409782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 513.181093] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 521.946555] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 527.331880] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 527.536270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 531.518258] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 531.727715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 533.989950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 533.993149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.994588] LustreError: Skipped 1 previous similar message [ 533.995244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 534.001961] Lustre: Skipped 1 previous similar message [ 539.815774] Lustre: server umount lustre-OST0000 complete [ 543.715972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 543.723776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.346502] LustreError: 15000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.440085] Lustre: server umount lustre-MDT0000 complete [ 553.145875] LustreError: 14980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693468 with bad export cookie 12655028913543305177 [ 553.148890] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 553.158509] LustreError: 14980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 553.510930] Lustre: server umount lustre-MDT0001 complete [ 556.991884] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 03:17:50 (1779693470) [ 564.607974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.246171] LustreError: 18848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.269374] LustreError: 18848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 565.338102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 570.268717] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 575.461882] LustreError: 18849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.479139] LustreError: 18849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 579.049883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.414930] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 589.166583] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 592.903829] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 599.901730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 606.735359] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 612.202261] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 620.004907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 620.020308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.035550] Lustre: Skipped 1 previous similar message [ 620.048973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 620.052967] Lustre: Skipped 6 previous similar messages [ 625.291636] Lustre: server umount lustre-OST0000 complete [ 630.755724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 630.759799] LustreError: Skipped 1 previous similar message [ 635.399170] Lustre: server umount lustre-MDT0000 complete [ 636.387323] LustreError: 18849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 636.414241] LustreError: 18849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 639.366753] LustreError: 18828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693554 with bad export cookie 12655028913543306059 [ 639.373785] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.383660] LustreError: 18828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 639.743200] Lustre: server umount lustre-MDT0001 complete [ 650.905622] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 653.865452] Key type lgssc unregistered [ 654.142177] LNet: 22023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.155036] LNetError: 22023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.177152] LNet: Removed LNI 192.168.204.115@tcp [ 655.368333] Key type .llcrypt unregistered [ 655.372248] Key type ._llcrypt unregistered [ 680.559161] Key type ._llcrypt registered [ 680.563211] Key type .llcrypt registered [ 680.702729] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 681.905982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 682.084439] alg: No test for adler32 (adler32-zlib) [ 683.318988] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 683.563104] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 685.280641] Key type lgssc registered [ 686.558337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 695.458367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 695.472050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.015662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.961814] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 707.555303] LustreError: 23419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.607357] LustreError: 23419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 711.718748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.125560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 716.991351] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 723.008525] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 727.428644] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 734.424399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 734.862469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 741.188596] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 742.076923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 747.306905] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 763.874182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 763.878696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 763.890793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 765.454645] Lustre: server umount lustre-OST0000 complete [ 773.098897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.106389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 773.119948] Lustre: Skipped 1 previous similar message [ 773.150594] Lustre: Skipped 1 previous similar message [ 778.210419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 778.215828] Lustre: Skipped 1 previous similar message [ 783.339240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 783.353853] Lustre: Skipped 1 previous similar message [ 784.352398] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 784.585910] Lustre: server umount lustre-MDT0000 complete [ 788.018647] LustreError: 23400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693703 with bad export cookie 867721992106540985 [ 788.024644] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 788.027979] LustreError: 23400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 788.389186] Lustre: server umount lustre-MDT0001 complete [ 798.352796] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 801.170804] Key type lgssc unregistered [ 801.533620] LNet: 27038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.545515] LNetError: 27038:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 801.561947] LNet: Removed LNI 192.168.204.115@tcp [ 802.497831] Key type .llcrypt unregistered [ 802.499355] Key type ._llcrypt unregistered [ 830.609834] Key type ._llcrypt registered [ 830.616852] Key type .llcrypt registered [ 830.780650] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 832.451388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 832.501579] alg: No test for adler32 (adler32-zlib) [ 833.621466] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 833.967497] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 835.688304] Key type lgssc registered [ 837.199635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 846.233828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 846.267663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.095140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 852.807532] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 858.611428] LustreError: 28435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 858.637453] LustreError: 28435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 863.297068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.761874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 868.957589] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 875.035812] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 880.040293] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 896.683041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 897.185280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 903.369396] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 908.642636] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 918.723961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 929.257386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.261261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 929.279331] Lustre: Skipped 1 previous similar message [ 929.281269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 929.306948] Lustre: Skipped 1 previous similar message [ 933.495947] Lustre: server umount lustre-OST0000 complete [ 939.492419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 939.497201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 939.501200] Lustre: Skipped 1 previous similar message [ 939.513241] Lustre: Skipped 1 previous similar message [ 943.673174] Lustre: server umount lustre-MDT0000 complete [ 944.622624] LustreError: 30941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 944.648381] LustreError: 30941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 947.229666] LustreError: 28416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693862 with bad export cookie 6845142935234752133 [ 947.243696] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 947.248178] LustreError: 28416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 947.628392] Lustre: server umount lustre-MDT0001 complete [ 958.658976] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 961.561456] Key type lgssc unregistered [ 961.916134] LNet: 31982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.926407] LNetError: 31982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.945619] LNet: Removed LNI 192.168.204.115@tcp [ 962.724289] Key type .llcrypt unregistered [ 962.732093] Key type ._llcrypt unregistered [ 986.282156] Key type ._llcrypt registered [ 986.284966] Key type .llcrypt registered [ 986.402818] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 987.599696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 987.625757] alg: No test for adler32 (adler32-zlib) [ 988.697418] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 988.943488] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 990.600152] Key type lgssc registered [ 991.800639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1001.469834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1001.489767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.189611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.537614] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1013.735498] LustreError: 33377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1013.768524] LustreError: 33377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1016.635674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.048991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1021.921532] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1028.180825] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1032.569920] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1039.279551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1039.581328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.563550] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1048.814925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1051.482056] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1061.865478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1061.877193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.887147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1063.394785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1063.404709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.420335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1065.850538] Lustre: server umount lustre-OST0000 complete [ 1073.633356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1073.646536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1073.666468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1073.676691] Lustre: Skipped 2 previous similar messages [ 1075.967434] Lustre: server umount lustre-MDT0000 complete [ 1079.750218] LustreError: 33358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693994 with bad export cookie 15791219654904121436 [ 1079.777483] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1080.111913] Lustre: server umount lustre-MDT0001 complete [ 1092.318474] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1095.400395] Key type lgssc unregistered [ 1095.735784] LNet: 36553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.748258] LNetError: 36553:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.768580] LNet: Removed LNI 192.168.204.115@tcp [ 1096.696444] Key type .llcrypt unregistered [ 1096.698275] Key type ._llcrypt unregistered [ 1120.460768] Key type ._llcrypt registered [ 1120.462854] Key type .llcrypt registered [ 1120.536848] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1121.843706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1121.923077] alg: No test for adler32 (adler32-zlib) [ 1122.982115] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 1123.301524] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1124.952171] Key type lgssc registered [ 1125.975171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1133.408961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1133.426828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.994428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1139.572604] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1145.318183] LustreError: 37950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1145.352038] LustreError: 37950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1149.173895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.565310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1153.789683] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1160.125642] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1164.447770] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1172.662727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1173.118300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1179.900961] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1180.336357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1186.444348] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1194.677841] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1194.978969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1195.000810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.002963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1196.006574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.010479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1196.013939] Lustre: Skipped 1 previous similar message [ 1199.012697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1199.018601] Lustre: Skipped 2 previous similar messages [ 1204.213123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1204.223250] Lustre: Skipped 2 previous similar messages [ 1208.288348] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1208.575603] Lustre: server umount lustre-OST0000 complete [ 1228.111250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.118342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1228.130465] Lustre: Skipped 1 previous similar message [ 1228.146426] Lustre: Skipped 2 previous similar messages [ 1234.409856] LustreError: 37950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.425233] LustreError: 37950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1234.694279] Lustre: server umount lustre-MDT0000 complete [ 1238.490885] LustreError: 37930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694153 with bad export cookie 3918656112562860060 [ 1238.504046] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1244.917553] Lustre: server umount lustre-MDT0001 complete [ 1255.932570] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1258.300868] Key type lgssc unregistered [ 1258.621539] LNet: 41301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.638784] LNetError: 41301:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.654479] LNet: Removed LNI 192.168.204.115@tcp [ 1259.464775] Key type .llcrypt unregistered [ 1259.466989] Key type ._llcrypt unregistered [ 1286.389746] Key type ._llcrypt registered [ 1286.391890] Key type .llcrypt registered [ 1286.485624] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1287.422929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1287.561474] alg: No test for adler32 (adler32-zlib) [ 1288.702400] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 1288.940955] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1290.656282] Key type lgssc registered [ 1291.863087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1299.270576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1299.314986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1300.848327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1305.416197] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1310.192115] LustreError: 42701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1310.214550] LustreError: 42701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1314.752276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.171776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1320.252886] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1326.702237] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1331.056837] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1337.799936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1338.158596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.598680] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1345.337938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1350.221347] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1360.868300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.872589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1360.879077] Lustre: Skipped 1 previous similar message [ 1362.111130] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1362.119638] Lustre: Skipped 2 previous similar messages [ 1363.574724] Lustre: server umount lustre-MDT0000 complete [ 1365.985533] LustreError: 42701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.010047] LustreError: 42701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1367.183827] LustreError: 42697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.728780] LustreError: 42682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694282 with bad export cookie 17057895321010975625 [ 1367.730279] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1367.746299] LustreError: 42682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1371.105274] LustreError: 42696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.106636] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1371.132252] Lustre: Skipped 1 previous similar message [ 1371.136705] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1377.249645] LustreError: 44672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.252036] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1377.270079] LustreError: 44672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1377.292621] Lustre: Skipped 2 previous similar messages [ 1382.369946] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1382.372266] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1382.396039] Lustre: Skipped 1 previous similar message [ 1382.728570] Lustre: server umount lustre-MDT0001 complete [ 1398.335302] Lustre: server umount lustre-OST0000 complete [ 1417.049348] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1420.038614] Key type lgssc unregistered [ 1420.375115] LNet: 46209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.387417] LNetError: 46209:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.403313] LNet: Removed LNI 192.168.204.115@tcp [ 1421.210320] Key type .llcrypt unregistered [ 1421.212266] Key type ._llcrypt unregistered [ 1448.162255] Key type ._llcrypt registered [ 1448.168753] Key type .llcrypt registered [ 1448.250985] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1449.576110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1449.590176] alg: No test for adler32 (adler32-zlib) [ 1450.650630] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 1450.921810] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1452.600165] Key type lgssc registered [ 1453.989399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1462.095477] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1462.117034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1477.667049] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1482.784263] LustreError: 47578:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 1509.472230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.964284] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1522.834375] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1688.197310] Lustre: server umount lustre-OST0000 complete [ 1706.298227] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1709.399732] Key type lgssc unregistered [ 1709.696896] LNet: 49122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.709855] LNetError: 49122:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.722538] LNet: Removed LNI 192.168.204.115@tcp [ 1710.461649] Key type .llcrypt unregistered [ 1710.464621] Key type ._llcrypt unregistered [ 1738.552745] Key type ._llcrypt registered [ 1738.554665] Key type .llcrypt registered [ 1738.674918] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1740.267881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1740.281387] alg: No test for adler32 (adler32-zlib) [ 1741.300533] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 1741.529412] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1743.192163] Key type lgssc registered [ 1744.560828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1754.313374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1754.347844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1755.842247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.026845] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1765.348474] LustreError: 50522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1765.372441] LustreError: 50522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1770.478129] LustreError: 50523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1772.115143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.719741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1778.866330] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1786.237286] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1791.343080] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1800.422495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1800.674534] LustreError: 52116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.875630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.347714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1808.037281] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1814.361129] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1818.218826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1818.219398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.253617] Lustre: Skipped 1 previous similar message [ 1823.386362] Lustre: server umount lustre-OST0000 complete [ 1828.833786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.836628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1828.841893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1828.864048] Lustre: Skipped 1 previous similar message [ 1833.958946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.974947] Lustre: Skipped 2 previous similar messages [ 1834.354557] Lustre: server umount lustre-MDT0000 complete [ 1838.218976] LustreError: 50502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694753 with bad export cookie 16801934162743694104 [ 1838.237723] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1838.627527] Lustre: server umount lustre-MDT0001 complete [ 1850.679909] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1854.170741] Key type lgssc unregistered [ 1854.571582] LNet: 53698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.581572] LNetError: 53698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.661303] LNet: Removed LNI 192.168.204.115@tcp [ 1856.829768] Key type .llcrypt unregistered [ 1856.831778] Key type ._llcrypt unregistered [ 1884.000386] Key type ._llcrypt registered [ 1884.007294] Key type .llcrypt registered [ 1884.138189] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1885.667496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1885.693076] alg: No test for adler32 (adler32-zlib) [ 1887.059852] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 1887.327462] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1888.976670] Key type lgssc registered [ 1890.671937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1900.378843] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1900.411286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1916.064404] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1921.187804] LustreError: 55069:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 1947.874942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.075798] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1961.882929] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1969.588640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.593886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.756890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1975.783698] LustreError: 55095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1975.821436] LustreError: 55095:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1977.590430] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1980.902614] LustreError: 55094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.962949] LustreError: 55991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.000974] LustreError: 55991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1988.070061] LustreError: 55993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.088333] LustreError: 55993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1988.776246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.169844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1994.396560] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2001.836471] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2006.468624] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2009.574329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2009.576099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.606593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2009.611280] Lustre: Skipped 1 previous similar message [ 2014.692031] LustreError: 55084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.703879] LustreError: 55084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2014.837935] Lustre: server umount lustre-OST0000 complete [ 2039.970251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.993332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2040.002873] Lustre: Skipped 2 previous similar messages [ 2040.023469] Lustre: Skipped 1 previous similar message [ 2045.410117] LustreError: 55997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.462096] Lustre: server umount lustre-MDT0000 complete [ 2049.753288] LustreError: 55076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694964 with bad export cookie 14587677635815600878 [ 2049.762904] LustreError: 55076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2049.765498] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2064.354252] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2064.690793] Lustre: server umount lustre-MDT0001 complete [ 2077.365513] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2081.729885] Key type lgssc unregistered [ 2082.288062] LNet: 58468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.305338] LNetError: 58468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2082.356094] LNet: Removed LNI 192.168.204.115@tcp [ 2083.463941] Key type .llcrypt unregistered [ 2083.470583] Key type ._llcrypt unregistered [ 2109.526780] Key type ._llcrypt registered [ 2109.528659] Key type .llcrypt registered [ 2109.649654] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2110.580668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2110.624026] alg: No test for adler32 (adler32-zlib) [ 2111.834434] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 2112.059263] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2113.752487] Key type lgssc registered [ 2115.013939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2124.900078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2124.915498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.594609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.496500] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2136.137899] LustreError: 59866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.164500] LustreError: 59866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2141.157023] LustreError: 59867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.665688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.956844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2148.252694] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2155.345500] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2160.540567] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2169.064673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2169.534328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.587444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2177.252513] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2183.791480] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2202.596128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2202.607839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.624137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2203.619113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2203.620819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2203.642478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.096679] Lustre: server umount lustre-OST0000 complete [ 2213.857794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.877714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2213.891256] Lustre: Skipped 2 previous similar messages [ 2218.416575] Lustre: server umount lustre-MDT0000 complete [ 2218.986046] LustreError: 59861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2219.017228] LustreError: 59861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2222.523948] LustreError: 59846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695137 with bad export cookie 5791445860443472040 [ 2222.552569] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.905629] Lustre: server umount lustre-MDT0001 complete [ 2234.080879] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2237.815573] Key type lgssc unregistered [ 2238.102250] LNet: 63067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.113771] LNetError: 63067:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.173842] LNet: Removed LNI 192.168.204.115@tcp [ 2239.222727] Key type .llcrypt unregistered [ 2239.226782] Key type ._llcrypt unregistered [ 2269.900348] Key type ._llcrypt registered [ 2269.902445] Key type .llcrypt registered [ 2270.038615] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2271.728830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.818621] alg: No test for adler32 (adler32-zlib) [ 2273.063266] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 2273.339224] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2275.128748] Key type lgssc registered [ 2276.792488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2287.053388] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 03:46:40 (1779695200) [ 2295.881084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2295.925385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.611514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.165231] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2307.049592] LustreError: 64756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.073969] LustreError: 64756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2311.439647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.933904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2316.324169] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2322.428635] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2326.467469] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2333.885270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2334.248989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.867426] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2342.455398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2346.934927] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2353.650348] Lustre: Failing over lustre-MDT0000 [ 2354.001572] Lustre: server umount lustre-MDT0000 complete [ 2354.885309] LustreError: 66270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.729723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2357.739528] LustreError: 66270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.747120] Lustre: Skipped 2 previous similar messages [ 2357.781161] LustreError: 66270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2359.953485] LustreError: 64751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.130535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2364.276538] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2364.435807] Lustre: *** cfs_fail_loc=135, val=0*** [ 2364.561522] LustreError: 67504:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2364.574877] LustreError: 67504:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.115@tcp: cfg command failed: rc = -2 [ 2364.582631] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2364.582631] [ 2364.591202] LustreError: MGC192.168.204.115@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2364.607482] LustreError: 67492:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2364.618307] LustreError: 67492:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2364.623856] LustreError: 67492:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2364.631648] LustreError: 67492:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2364.665866] Lustre: server umount lustre-MDT0000 complete [ 2364.675567] LustreError: 67492:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2365.085952] LustreError: 66270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.114294] LustreError: 66270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2372.346933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.544418] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2372.557574] LustreError: 64747:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2372.564695] LustreError: 67851:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGC192.168.204.115@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000f16ec2cc) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2372.887321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2372.969445] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2375.253498] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2378.227326] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2378.270813] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2378.322059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2378.443318] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2387.426219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2387.432457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.450118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2388.454213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2388.476638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.837572] Lustre: server umount lustre-OST0000 complete [ 2398.689409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.702642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2398.708095] Lustre: Skipped 1 previous similar message [ 2398.723300] Lustre: Skipped 2 previous similar messages [ 2403.417358] Lustre: server umount lustre-MDT0000 complete [ 2403.813000] LustreError: 64750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.834526] LustreError: 64750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2407.774885] LustreError: 64735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695322 with bad export cookie 17116499105682126017 [ 2407.783463] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2407.794164] LustreError: 64735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2408.092920] Lustre: server umount lustre-MDT0001 complete [ 2420.421398] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2423.582200] Key type lgssc unregistered [ 2423.965412] LNet: 69250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2423.976047] LNetError: 69250:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2423.992167] LNet: Removed LNI 192.168.204.115@tcp [ 2424.889507] Key type .llcrypt unregistered [ 2424.891645] Key type ._llcrypt unregistered [ 2451.126150] Key type ._llcrypt registered [ 2451.129211] Key type .llcrypt registered [ 2451.279923] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2452.636590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2452.654270] alg: No test for adler32 (adler32-zlib) [ 2453.717788] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 2453.974546] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2455.673365] Key type lgssc registered [ 2457.104500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2464.884684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2464.907609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.344719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.730992] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2476.519131] LustreError: 70647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.542823] LustreError: 70647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2479.441209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.920834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2485.046534] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2491.508693] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2496.208671] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2504.158850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2504.523200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2511.104496] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2511.829282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2518.705261] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2524.759664] Lustre: Failing over lustre-MDT0000 [ 2525.122920] Lustre: server umount lustre-MDT0000 complete [ 2525.873940] LustreError: 70642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.178938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2526.192057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.201766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.202631] LustreError: 70647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.237269] LustreError: 70647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2530.979970] LustreError: 70642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.804282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2535.969425] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2536.088462] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (not set up) [ 2536.218474] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2536.375749] LustreError: 73396:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2536.383576] LustreError: 73396:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.115@tcp: cfg command failed: rc = -9 [ 2536.397793] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2536.397793] [ 2536.408711] LustreError: MGC192.168.204.115@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2536.433510] LustreError: 73384:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2536.451398] LustreError: 73384:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 2536.466496] LustreError: 73384:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2536.475946] LustreError: 73384:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2536.511729] Lustre: server umount lustre-MDT0000 complete [ 2536.517419] LustreError: 73384:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2537.445688] LustreError: 70646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.477131] LustreError: 70646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2544.174069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.389220] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2544.408982] LustreError: 70638:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2544.730507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.797376] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2546.337711] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2549.633660] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2549.745898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2549.766779] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2549.818217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2559.979745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.984624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2560.000246] Lustre: Skipped 2 previous similar messages [ 2563.978668] Lustre: server umount lustre-OST0000 complete [ 2570.209845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2570.211476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.238398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2570.256375] Lustre: Skipped 1 previous similar message [ 2570.262542] Lustre: Skipped 1 previous similar message [ 2575.064390] Lustre: server umount lustre-MDT0000 complete [ 2575.335526] LustreError: 72749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.380559] LustreError: 72749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2581.358513] LustreError: 70627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695496 with bad export cookie 5072071465044212793 [ 2581.374169] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2581.377326] LustreError: 70627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2582.096815] Lustre: server umount lustre-MDT0001 complete [ 2595.599687] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2598.829217] Key type lgssc unregistered [ 2599.138368] LNet: 75143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2599.143225] LNetError: 75143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2599.152267] LNet: Removed LNI 192.168.204.115@tcp [ 2599.796708] Key type .llcrypt unregistered [ 2599.800787] Key type ._llcrypt unregistered [ 2614.361645] Key type ._llcrypt registered [ 2614.364435] Key type .llcrypt registered [ 2614.507570] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2614.861614] Key type .llcrypt unregistered [ 2614.868543] Key type ._llcrypt unregistered [ 2639.589250] Key type ._llcrypt registered [ 2639.594609] Key type .llcrypt registered [ 2639.712146] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2640.671471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2640.695651] alg: No test for adler32 (adler32-zlib) [ 2641.822313] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 2642.070498] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2643.760396] Key type lgssc registered [ 2645.186450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2653.515619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2653.535844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2655.212382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.228361] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2665.447505] LustreError: 77015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.469802] LustreError: 77015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2670.039197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2670.431599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2674.993914] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2681.395117] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2685.032712] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2692.572399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2692.857696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.455759] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2700.030448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2705.453946] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2716.641528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2716.661619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.692873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2720.742992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2720.743464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2720.759792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.234799] Lustre: server umount lustre-OST0000 complete [ 2726.882943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2726.903036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.923377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2726.926870] Lustre: Skipped 1 previous similar message [ 2730.978945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2730.982185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.677447] Lustre: server umount lustre-MDT0000 complete [ 2736.110516] LustreError: 77990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2736.127645] LustreError: 77990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2736.646096] LustreError: 79166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695651 with bad export cookie 5986893919342437380 [ 2736.648285] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2736.655216] LustreError: 79166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2737.094467] Lustre: server umount lustre-MDT0001 complete [ 2749.238714] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2752.374272] Key type lgssc unregistered [ 2752.702982] LNet: 80192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2752.728315] LNetError: 80192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2752.750651] LNet: Removed LNI 192.168.204.115@tcp [ 2753.693601] Key type .llcrypt unregistered [ 2753.695979] Key type ._llcrypt unregistered [ 2779.058785] Key type ._llcrypt registered [ 2779.061923] Key type .llcrypt registered [ 2779.195121] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2780.342623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2780.439204] alg: No test for adler32 (adler32-zlib) [ 2781.641719] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 2781.887362] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2783.585328] Key type lgssc registered [ 2785.080267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2793.886938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2793.913356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.579836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2800.512626] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2805.733811] LustreError: 81591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.760163] LustreError: 81591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2806.759669] LustreError: 81592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.813960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2810.226610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2814.860802] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2821.433605] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2826.096992] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2835.262390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2835.679513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.851975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2843.627759] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2849.656241] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2860.516216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2860.524448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.557540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2862.322558] Lustre: server umount lustre-OST0000 complete [ 2866.657542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2866.674541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.696377] Lustre: Skipped 1 previous similar message [ 2866.708242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2866.711039] Lustre: Skipped 1 previous similar message [ 2870.760090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.761988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.834436] Lustre: server umount lustre-MDT0000 complete [ 2875.883275] LustreError: 81592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.907073] LustreError: 81592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2876.504875] LustreError: 81570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695791 with bad export cookie 11746370092779706149 [ 2876.508630] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2876.519934] LustreError: 81570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2876.786725] Lustre: server umount lustre-MDT0001 complete [ 2888.373445] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2891.949707] Key type lgssc unregistered [ 2892.257555] LNet: 84768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2892.265256] LNetError: 84768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2892.278606] LNet: Removed LNI 192.168.204.115@tcp [ 2893.928808] Key type .llcrypt unregistered [ 2893.932028] Key type ._llcrypt unregistered [ 2920.200333] Key type ._llcrypt registered [ 2920.203465] Key type .llcrypt registered [ 2920.355565] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2921.685731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2921.790498] alg: No test for adler32 (adler32-zlib) [ 2922.926830] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 2923.223524] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2924.969823] Key type lgssc registered [ 2926.564859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2935.160533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2935.174954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.735935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.006397] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2946.041939] LustreError: 86166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.079379] LustreError: 86166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2951.137230] LustreError: 86165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2952.200256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2952.642581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2957.051386] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2963.085996] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2967.772519] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2974.782534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2975.101438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.255082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2981.918870] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2988.452681] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3002.850214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3002.862477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.885308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3006.949961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3006.952710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3006.961833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3008.476322] Lustre: server umount lustre-OST0000 complete [ 3014.117816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3014.131996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.152187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3014.159515] Lustre: Skipped 1 previous similar message [ 3017.185361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.186822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3017.220558] Lustre: Skipped 1 previous similar message [ 3019.116295] Lustre: server umount lustre-MDT0000 complete [ 3022.305259] LustreError: 86166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.327535] LustreError: 86166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3022.999893] LustreError: 86146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695938 with bad export cookie 9719146283451017786 [ 3023.002066] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3023.015634] LustreError: 86146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3023.629278] Lustre: server umount lustre-MDT0001 complete [ 3037.120658] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3040.281359] Key type lgssc unregistered [ 3040.673704] LNet: 89364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3040.683674] LNetError: 89364:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3040.717907] LNet: Removed LNI 192.168.204.115@tcp [ 3041.503323] Key type .llcrypt unregistered [ 3041.505494] Key type ._llcrypt unregistered [ 3067.758450] Key type ._llcrypt registered [ 3067.761564] Key type .llcrypt registered [ 3067.856042] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3068.783104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3068.856167] alg: No test for adler32 (adler32-zlib) [ 3069.965983] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 3070.188686] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3071.896810] Key type lgssc registered [ 3072.974978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3080.568328] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3080.589701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.160362] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3101.280207] LustreError: 90731:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 3128.032380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3134.560590] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3140.693307] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3146.739700] Lustre: server umount lustre-OST0000 complete [ 3155.823737] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 04:01:09 (1779696069) [ 3163.673080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.268104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3169.539567] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3169.763311] LustreError: 92272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.786049] LustreError: 92272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3174.883328] LustreError: 92273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3179.375489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3179.841357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3184.996984] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3191.113815] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3195.179542] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3202.731660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3203.242501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3209.388629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3210.178936] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3217.194709] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3230.184061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.211395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3230.214157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3230.242612] Lustre: Skipped 1 previous similar message [ 3234.885987] Lustre: server umount lustre-OST0000 complete [ 3240.420097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.443232] Lustre: Skipped 1 previous similar message [ 3240.452952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3245.468116] Lustre: server umount lustre-MDT0000 complete [ 3245.542792] LustreError: 92273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3245.586482] LustreError: 92273:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3250.060653] LustreError: 92252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696165 with bad export cookie 11136325420238241484 [ 3250.064716] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3250.069779] LustreError: 92252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3250.486895] Lustre: server umount lustre-MDT0001 complete [ 3261.816996] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3264.849542] Key type lgssc unregistered [ 3265.212143] LNet: 95607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3265.216289] LNetError: 95607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3265.229564] LNet: Removed LNI 192.168.204.115@tcp [ 3266.125610] Key type .llcrypt unregistered [ 3266.127861] Key type ._llcrypt unregistered [ 3308.505105] Key type ._llcrypt registered [ 3308.508251] Key type .llcrypt registered [ 3308.586912] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3309.697071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3309.794128] alg: No test for adler32 (adler32-zlib) [ 3310.872316] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 3311.157889] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3312.856591] Key type lgssc registered [ 3313.970604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3320.943280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3320.979302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.362945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.462685] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3332.582679] LustreError: 97778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.595944] LustreError: 97778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3335.126297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.482859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3340.470180] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3347.942771] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3352.427859] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3359.910095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3360.241936] LustreError: 99371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3360.493866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3367.240521] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3368.953515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3373.676226] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3384.292421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.306809] Lustre: Skipped 1 previous similar message [ 3384.309310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3389.383675] Lustre: server umount lustre-OST0000 complete [ 3394.530233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.534196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3394.541171] Lustre: Skipped 1 previous similar message [ 3394.555232] Lustre: Skipped 2 previous similar messages [ 3399.653312] LustreError: 97778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3399.732121] Lustre: server umount lustre-MDT0000 complete [ 3403.181566] LustreError: 97760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696318 with bad export cookie 8933139284938870994 [ 3403.184049] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3403.190493] LustreError: 97760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3403.641979] Lustre: server umount lustre-MDT0001 complete [ 3414.299567] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3417.199466] Key type lgssc unregistered [ 3417.552695] LNet: 100953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3417.567499] LNetError: 100953:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3417.585400] LNet: Removed LNI 192.168.204.115@tcp [ 3418.400649] Key type .llcrypt unregistered [ 3418.402322] Key type ._llcrypt unregistered [ 3437.607835] Key type ._llcrypt registered [ 3437.611259] Key type .llcrypt registered [ 3437.706527] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3438.896293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3438.955696] alg: No test for adler32 (adler32-zlib) [ 3440.047395] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 3440.301488] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3442.016155] Key type lgssc registered [ 3443.290473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3451.614301] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3451.642541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3467.232678] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3472.352359] LustreError: 102183:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 3499.040179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3505.367869] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3510.658585] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3516.760645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.073320] LustreError: MGC192.168.204.115@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3517.086478] LustreError: 103094:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3517.093173] LustreError: 103094:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3517.100539] LustreError: 103094:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3517.174898] Lustre: server umount lustre-MDT0000 complete [ 3517.181198] LustreError: 103094:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3534.582766] Lustre: server umount lustre-OST0000 complete [ 3547.820307] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3558.298466] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3596.425972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.791949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3596.851515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3596.994825] Lustre: lustre-MDT0000: new disk, initializing [ 3597.148416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3597.169735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3602.535193] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3614.642558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.792212] Lustre: 105849:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3614.909774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3614.920052] Lustre: Skipped 1 previous similar message [ 3615.047361] Lustre: lustre-MDT0001: new disk, initializing [ 3615.152756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3615.199647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3615.208716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3619.807399] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3626.984359] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3630.764279] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3636.826862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3637.079727] Lustre: lustre-OST0000: new disk, initializing [ 3637.083542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3637.090504] Lustre: 107595:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3637.188903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.977872] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3643.444122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3643.453983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3643.601774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3650.865446] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3655.362982] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3655.566561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3660.333961] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3660.610598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3664.359911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3664.363329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.366983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3664.420677] Lustre: Skipped 1 previous similar message [ 3668.716962] Lustre: server umount lustre-OST0000 complete [ 3674.599545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.606411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3674.635337] Lustre: Skipped 1 previous similar message [ 3674.636326] Lustre: Skipped 1 previous similar message [ 3679.716494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3679.720503] Lustre: Skipped 2 previous similar messages [ 3684.837404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3684.848857] Lustre: Skipped 1 previous similar message [ 3686.368141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3686.482684] Lustre: server umount lustre-MDT0000 complete [ 3691.573255] LustreError: 105843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696606 with bad export cookie 7355248398746417029 [ 3691.579124] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3691.592758] LustreError: 105843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3692.213401] Lustre: server umount lustre-MDT0001 complete [ 3701.389721] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 04:10:14 (1779696614) [ 3705.580180] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3724.350585] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3734.585395] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3773.700560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3774.090322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3774.133218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3774.243202] Lustre: lustre-MDT0000: new disk, initializing [ 3774.422437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3774.459517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3779.534848] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3791.697256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3791.859982] Lustre: 112315:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3791.908325] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3791.911083] Lustre: Skipped 1 previous similar message [ 3791.991254] Lustre: lustre-MDT0001: new disk, initializing [ 3792.077148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3792.103851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3792.126554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3796.751289] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3805.156835] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3809.299883] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3816.574899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3816.884313] Lustre: lustre-OST0000: new disk, initializing [ 3816.888483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3816.905806] Lustre: 114059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3817.015487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3823.755925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3823.778357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3823.808349] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3823.906970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3832.224969] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3836.849188] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3837.071468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3841.565716] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3841.760575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3844.585902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3844.603334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3844.608282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3844.612221] Lustre: Skipped 1 previous similar message [ 3844.626961] LustreError: Skipped 1 previous similar message [ 3849.826945] Lustre: server umount lustre-OST0000 complete [ 3854.822437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.833158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3854.837042] Lustre: Skipped 4 previous similar messages [ 3860.400344] Lustre: server umount lustre-MDT0000 complete [ 3864.311699] LustreError: 112309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696779 with bad export cookie 7355248398746417911 [ 3864.313084] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3864.328200] LustreError: 112309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3864.797266] Lustre: server umount lustre-MDT0001 complete [ 3873.038815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.370394] LustreError: 116131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3873.394156] LustreError: 116131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3873.466278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3878.025588] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3878.887240] LustreError: 116132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.984469] LustreError: 116131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3887.269936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.353582] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3898.467379] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3902.765536] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3910.092417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3917.404293] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3924.666115] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3932.352438] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3938.273164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3938.282149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.300114] Lustre: Skipped 1 previous similar message [ 3938.310452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3938.317507] Lustre: Skipped 2 previous similar messages [ 3943.510775] Lustre: server umount lustre-OST0000 complete [ 3949.026724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3949.038048] LustreError: Skipped 1 previous similar message [ 3949.041761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.056606] Lustre: Skipped 1 previous similar message [ 3962.336136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3962.492221] Lustre: server umount lustre-MDT0000 complete [ 3964.906774] LustreError: 117097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3964.922659] LustreError: 117097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3966.234790] LustreError: 116113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696881 with bad export cookie 7355248398746418793 [ 3966.244254] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3966.251049] LustreError: 116113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3966.664537] Lustre: server umount lustre-MDT0001 complete [ 3977.623865] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3981.214858] Key type lgssc unregistered [ 3981.594303] LNet: 119405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3981.601327] LNetError: 119405:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3981.624512] LNet: Removed LNI 192.168.204.115@tcp [ 3982.655228] Key type .llcrypt unregistered [ 3982.660731] Key type ._llcrypt unregistered [ 4004.396425] Key type ._llcrypt registered [ 4004.399386] Key type .llcrypt registered [ 4004.539959] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4021.239139] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4022.424635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4022.444320] alg: No test for adler32 (adler32-zlib) [ 4023.746950] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 4024.156513] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4025.880210] Key type lgssc registered [ 4027.174321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4065.594503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4065.624744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.922469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4066.949872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4067.040324] Lustre: lustre-MDT0000: new disk, initializing [ 4067.128356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4067.166568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4071.807214] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4082.021813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4082.139571] Lustre: 122525:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4082.176915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4082.179965] Lustre: Skipped 1 previous similar message [ 4082.272829] Lustre: lustre-MDT0001: new disk, initializing [ 4082.386624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4082.422225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4082.436689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4087.155592] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4095.406822] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4099.261990] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4106.915298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4107.271517] Lustre: lustre-OST0000: new disk, initializing [ 4107.277951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4107.288793] Lustre: 124269:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4107.380729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.610390] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4113.983839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4114.003293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4114.110342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4121.454940] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4126.160148] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4126.380688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4130.546265] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4130.737318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4134.891950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.893940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4134.902900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4134.918784] Lustre: Skipped 1 previous similar message [ 4134.959996] LustreError: Skipped 1 previous similar message [ 4138.980498] Lustre: server umount lustre-OST0000 complete [ 4144.099655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4144.105159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.119971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.131956] Lustre: Skipped 1 previous similar message [ 4145.125305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4145.129638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.186678] Lustre: server umount lustre-MDT0000 complete [ 4152.519492] LustreError: 122517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697067 with bad export cookie 3704658755658953590 [ 4152.522223] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4152.532697] LustreError: 122517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4152.935610] Lustre: server umount lustre-MDT0001 complete [ 4161.073376] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 04:17:54 (1779697074) [ 4167.624290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4168.019916] LustreError: 126535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4168.032382] LustreError: 126535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4168.063216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.724563] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4173.289088] LustreError: 126536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.407258] LustreError: 126535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4179.301702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4179.691558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4184.475558] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4189.778400] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4192.493694] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4195.297283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4195.302610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.312648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4195.317687] Lustre: Skipped 1 previous similar message [ 4200.347514] Lustre: server umount lustre-MDT0000 complete [ 4203.491096] LustreError: 127247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.523255] LustreError: 127247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4203.667358] LustreError: 127877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697118 with bad export cookie 3704658755658954472 [ 4203.673790] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4203.681341] LustreError: 127877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4204.026906] Lustre: server umount lustre-MDT0001 complete [ 4211.805439] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 04:18:45 (1779697125) [ 4217.884879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4233.440647] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4238.560874] LustreError: 128722:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 4264.352559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4271.155241] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4277.880116] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4280.337276] Lustre: server umount lustre-OST0000 complete [ 4288.008982] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 04:20:01 (1779697201) [ 4293.778256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.093086] LustreError: 130055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4294.185774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4297.899437] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4304.370506] LustreError: 130056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.398526] LustreError: 130056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4306.702896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.740707] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4317.440984] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4322.433956] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4330.973509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4331.316682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4331.324099] Lustre: Skipped 1 previous similar message [ 4338.472812] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4344.247482] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4353.516518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4353.519921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.537725] Lustre: Skipped 1 previous similar message [ 4353.548647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4353.555381] Lustre: Skipped 2 previous similar messages [ 4359.268718] Lustre: server umount lustre-MDT0000 complete [ 4360.162668] LustreError: 130055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.177954] LustreError: 130055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4362.827233] LustreError: 130037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697277 with bad export cookie 3704658755658955137 [ 4362.829935] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4362.842107] LustreError: 130037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4363.188641] Lustre: server umount lustre-MDT0001 complete [ 4377.057494] Lustre: server umount lustre-OST0000 complete [ 4384.898645] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 04:21:38 (1779697298) [ 4391.941574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4392.606752] LustreError: 133369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4396.976932] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4405.820609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4406.303409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4406.307639] Lustre: Skipped 1 previous similar message [ 4410.737300] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4417.410289] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4421.469367] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4428.673289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4435.394881] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4437.373937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4441.431922] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4446.057475] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4446.258737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4450.776583] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4450.959682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4456.941964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4456.949625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4456.968173] Lustre: Skipped 2 previous similar messages [ 4456.984775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4456.990730] Lustre: Skipped 3 previous similar messages [ 4467.168162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4467.456849] Lustre: server umount lustre-OST0000 complete [ 4472.801838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4472.826422] LustreError: Skipped 1 previous similar message [ 4473.316511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4473.326177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4473.334701] Lustre: Skipped 2 previous similar messages [ 4473.345635] Lustre: Skipped 5 previous similar messages [ 4486.112274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4486.425934] Lustre: server umount lustre-MDT0000 complete [ 4488.677173] LustreError: 133370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4488.694830] LustreError: 133370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4490.422970] LustreError: 133351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697405 with bad export cookie 3704658755658956208 [ 4490.428035] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4490.435814] LustreError: 133351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4490.710971] Lustre: server umount lustre-MDT0001 complete [ 4502.335083] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 04:23:34 (1779697414) [ 4511.158764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4526.752731] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4531.872534] LustreError: 137158:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 4557.600460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4557.618022] Lustre: Skipped 1 previous similar message [ 4566.346619] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4574.159938] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4582.625390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4583.657024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4589.999812] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4602.252460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.975552] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4616.407079] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4621.244117] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4627.078563] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4627.346411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4633.122052] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4633.385248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4638.690217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4638.691739] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4638.701636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4638.701648] Lustre: Skipped 5 previous similar messages [ 4638.749049] Lustre: Skipped 1 previous similar message [ 4642.429833] Lustre: server umount lustre-MDT0000 complete [ 4643.818422] LustreError: 138082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.833738] LustreError: 138082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4646.736899] LustreError: 137166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697561 with bad export cookie 3704658755658957230 [ 4646.744045] LustreError: 137166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4646.756203] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4647.071337] Lustre: server umount lustre-MDT0001 complete [ 4661.684431] Lustre: server umount lustre-OST0000 complete [ 4671.826236] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 04:26:25 (1779697585) [ 4678.911624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4694.497348] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4699.618413] LustreError: 140982:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 4733.158936] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4741.353117] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4749.150608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4749.738037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4754.829430] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4765.258672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4771.173266] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4778.076264] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4783.977159] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4793.051818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4793.382635] Lustre: lustre-OST0001: new disk, initializing [ 4793.387610] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4793.396225] Lustre: 143517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4798.553946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4798.566145] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4798.694852] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4801.715712] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4812.163601] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4817.954638] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4818.273357] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4823.268410] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4823.492959] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4826.605168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4826.617868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4826.641507] Lustre: Skipped 1 previous similar message [ 4826.655671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4826.662043] Lustre: Skipped 2 previous similar messages [ 4831.738714] Lustre: server umount lustre-OST0000 complete [ 4851.168265] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4851.460698] Lustre: server umount lustre-OST0001 complete [ 4863.008747] Lustre: server umount lustre-MDT0000 complete [ 4867.603620] LustreError: 140990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697782 with bad export cookie 3704658755658958182 [ 4867.630674] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4867.637036] LustreError: 140990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4868.033438] Lustre: server umount lustre-MDT0001 complete [ 4898.208895] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 04:30:10 (1779697810) [ 4900.415688] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4902.465106] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4904.781579] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 04:30:17 (1779697817) [ 4912.942490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4913.445847] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4913.479813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4913.742837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4913.745774] Lustre: Skipped 6 previous similar messages [ 4919.646773] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4930.712974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4930.876260] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4930.942234] Lustre: 146936:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4937.014904] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4944.945426] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4950.035793] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4958.783561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4959.034830] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4967.209742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4967.853667] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4975.245731] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4981.511064] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4981.848492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4987.999408] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4988.377489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4993.011663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4993.033603] LustreError: Skipped 4 previous similar messages [ 4993.034873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4993.037231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4993.037238] Lustre: Skipped 12 previous similar messages [ 4993.075648] Lustre: Skipped 6 previous similar messages [ 4997.529662] Lustre: server umount lustre-OST0000 complete [ 5024.450512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5033.272881] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.292107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.297429] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5033.947447] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5041.872343] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5048.688100] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5049.266915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5055.412919] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5055.795180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5060.983861] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92ac1001c800.ost_server_uuid 50 [ 5062.870664] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff92ac1001c800.ost_server_uuid in FULL state after 0 sec [ 5063.114585] LustreError: 149077:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.15@tcp) returned error from blocking AST (req@000000002252a16a x1866147497780864 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000047c34ea4/0x336997829917cec0 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.15@tcp remote: 0xc4d9edcb34111e49 expref: 5 pid: 148041 timeout: 5163 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5063.168426] LustreError: lustre-MDT0000: A client on nid 192.168.204.15@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5063.187877] LustreError: 146931:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.15@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000047c34ea4/0x336997829917cec0 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.15@tcp remote: 0xc4d9edcb34111e49 expref: 6 pid: 148041 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5075.308447] Lustre: server umount lustre-OST0000 complete [ 5087.867262] Lustre: server umount lustre-MDT0000 complete [ 5089.764787] LustreError: 148040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5089.797763] LustreError: 148040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 5093.916210] LustreError: 146926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698008 with bad export cookie 3704658755658959365 [ 5093.942483] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5093.948739] LustreError: 146926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5100.615766] Lustre: server umount lustre-MDT0001 complete [ 5115.716848] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5120.210918] Key type lgssc unregistered [ 5120.583042] LNet: 152419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5120.601208] LNetError: 152419:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5120.635645] LNet: Removed LNI 192.168.204.115@tcp [ 5121.954170] Key type .llcrypt unregistered [ 5121.956991] Key type ._llcrypt unregistered [ 5152.745631] Key type ._llcrypt registered [ 5152.749196] Key type .llcrypt registered [ 5152.898931] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5154.341479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5154.378179] alg: No test for adler32 (adler32-zlib) [ 5155.740646] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 5156.092214] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5157.824415] Key type lgssc registered [ 5159.748083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5171.776971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5171.826449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5173.637193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5178.697948] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5182.947735] LustreError: 153817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5182.968064] LustreError: 153817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5188.067616] LustreError: 153818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5188.719312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5189.121496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5194.181487] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5201.235659] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5205.356922] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5214.655174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5215.135578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.247253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5222.919648] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5229.548988] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5239.992589] Lustre: Failing over lustre-MDT0000 [ 5240.294195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5240.304865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5240.336770] LustreError: 153817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.488126] Lustre: server umount lustre-MDT0000 complete [ 5242.553663] LustreError: 153813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5244.907822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.926079] Lustre: Skipped 1 previous similar message [ 5250.026000] LustreError: 153812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5250.048669] LustreError: 153812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5253.153096] LustreError: 153799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.15@tcp arrived at 1779698168 with bad export cookie 17265857368116213277 [ 5260.258618] Lustre: 153128:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698159/real 1779698159] req@0000000006b977d7 x1866148684526336/t0(0) o400->MGC192.168.204.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779698175 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5260.268515] LustreError: 153818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5260.289890] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5260.327806] LustreError: 153818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5262.860900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.515076] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xef9ca5cc4120b686 [ 5270.537387] Lustre: MGC192.168.204.115@tcp: Connection restored to 0@lo (at 0@lo) [ 5270.894420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5270.973505] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5276.154158] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5276.165660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5277.445388] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5280.324559] Lustre: lustre-MDT0000: Denying connection for new client e67db1be-a21d-4a00-a1fe-7dacb1ee4921 (at 192.168.204.15@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5290.978817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5290.997911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5291.036122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5295.452819] Lustre: server umount lustre-OST0000 complete [ 5300.545836] LustreError: 157197:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5300.563412] Lustre: 156561:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5300.578125] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5300.598225] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5300.626826] LustreError: 156561:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d7c60d70 x1866148684546688/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5300.649944] LustreError: 156561:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5300.673227] LustreError: 156561:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5300.687305] Lustre: lustre-MDT0000: Recovery over after 0:24, of 2 clients 0 recovered and 2 were evicted. [ 5301.266839] Lustre: server umount lustre-MDT0000 complete [ 5301.731340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5301.747797] Lustre: Skipped 1 previous similar message [ 5301.771057] LustreError: 153812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5301.796519] LustreError: 153812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5306.325623] LustreError: 153126:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5306.329689] LustreError: 153798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698221 with bad export cookie 17265857368116213382 [ 5306.330565] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5306.373570] LustreError: 153798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5313.042479] Lustre: server umount lustre-MDT0001 complete [ 5326.811635] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5330.713754] Key type lgssc unregistered [ 5331.035576] LNet: 157934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5331.048533] LNetError: 157934:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5331.078888] LNet: Removed LNI 192.168.204.115@tcp [ 5332.140331] Key type .llcrypt unregistered [ 5332.142697] Key type ._llcrypt unregistered [ 5358.377860] Key type ._llcrypt registered [ 5358.384356] Key type .llcrypt registered [ 5358.495793] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5360.614521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5360.774303] alg: No test for adler32 (adler32-zlib) [ 5362.121501] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 5362.665347] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5364.432363] Key type lgssc registered [ 5366.050555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5376.614017] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5376.664776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5378.332402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5384.144332] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5388.774428] LustreError: 159347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5388.800771] LustreError: 159347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5393.909357] LustreError: 159346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5394.803778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5395.129313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5400.291480] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5406.430516] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5410.362635] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5418.421887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5418.833319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.931309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5426.050292] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5433.667874] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5443.947988] Lustre: server umount lustre-OST0000 complete [ 5449.704761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5449.706981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5449.716097] Lustre: Skipped 1 previous similar message [ 5449.733414] Lustre: Skipped 1 previous similar message [ 5454.821123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5454.834283] Lustre: Skipped 1 previous similar message [ 5459.078371] Lustre: server umount lustre-MDT0000 complete [ 5459.938622] Lustre: 158658:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698359/real 1779698359] req@00000000d16c2a4a x1866148900531712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779698375 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5459.940260] LustreError: 160458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.979916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.027601] LustreError: 160458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5462.048236] Lustre: 158658:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698361/real 1779698361] req@0000000046d7d9ec x1866148900532096/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779698377 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5463.349715] LustreError: 159328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698378 with bad export cookie 7747318005791405387 [ 5463.355857] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5463.360054] LustreError: 159328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5464.121319] Lustre: server umount lustre-MDT0001 complete [ 5475.781200] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5478.639907] Key type lgssc unregistered [ 5478.962642] LNet: 162523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5478.976141] LNetError: 162523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5478.994349] LNet: Removed LNI 192.168.204.115@tcp [ 5479.734647] Key type .llcrypt unregistered [ 5479.737978] Key type ._llcrypt unregistered [ 5514.707990] Key type ._llcrypt registered [ 5514.710523] Key type .llcrypt registered [ 5514.808980] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5515.951040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5516.014955] alg: No test for adler32 (adler32-zlib) [ 5517.191919] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 5517.456497] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5519.201051] Key type lgssc registered [ 5520.437383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5529.224480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5529.262375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.881718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5535.586050] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5540.328607] LustreError: 164208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5540.347429] LustreError: 164208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5544.485305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5544.882539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5549.433853] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5554.744226] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5558.278575] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5565.697613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5566.234355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5573.136667] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5575.475219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5578.912676] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5586.680535] Lustre: server umount lustre-OST0000 complete [ 5591.018103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.033115] Lustre: Skipped 1 previous similar message [ 5591.038404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5591.059578] Lustre: Skipped 1 previous similar message [ 5596.134292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5600.392506] Lustre: server umount lustre-MDT0000 complete [ 5601.251483] LustreError: 164920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.276673] LustreError: 164920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5605.038081] LustreError: 164190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698520 with bad export cookie 13930731716193528247 [ 5605.043880] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5605.056121] LustreError: 164190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5615.843579] Lustre: server umount lustre-MDT0001 complete [ 5627.207757] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5630.408895] Key type lgssc unregistered [ 5630.708161] LNet: 167385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5630.715996] LNetError: 167385:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5630.733534] LNet: Removed LNI 192.168.204.115@tcp [ 5631.707637] Key type .llcrypt unregistered [ 5631.711096] Key type ._llcrypt unregistered [ 5655.108789] Key type ._llcrypt registered [ 5655.114257] Key type .llcrypt registered [ 5655.232794] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5656.489224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5656.539166] alg: No test for adler32 (adler32-zlib) [ 5657.633358] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 5657.862976] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5659.520347] Key type lgssc registered [ 5660.874785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5674.159885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5674.178414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.602792] Lustre: *** cfs_fail_loc=135, val=0*** [ 5675.703788] LustreError: 168892:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5675.715988] LustreError: 168892:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.115@tcp: cfg command failed: rc = -2 [ 5675.742563] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5675.742563] [ 5675.764076] LustreError: MGC192.168.204.115@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5675.797240] LustreError: 168853:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5675.815199] LustreError: 168853:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5675.824866] LustreError: 168853:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5675.827853] LustreError: 168853:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5675.923395] Lustre: server umount lustre-MDT0000 complete [ 5675.927661] LustreError: 168853:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5683.489883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5684.087923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.674832] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5694.435845] LustreError: 169264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.454936] LustreError: 169264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5699.554032] LustreError: 169265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5700.398981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.000071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5705.757460] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5712.073924] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5716.488655] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5719.127576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.140873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5724.140106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5724.145928] Lustre: Skipped 2 previous similar messages [ 5725.157762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5725.178034] Lustre: Skipped 1 previous similar message [ 5725.733757] Lustre: server umount lustre-MDT0000 complete [ 5729.896520] LustreError: 169246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698644 with bad export cookie 2877300648921416692 [ 5729.905911] LustreError: 169246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5729.906504] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5730.354758] Lustre: server umount lustre-MDT0001 complete [ 5741.148232] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5743.801887] Key type lgssc unregistered [ 5744.103726] LNet: 171342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5744.109984] LNetError: 171342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5744.129625] LNet: Removed LNI 192.168.204.115@tcp [ 5744.910535] Key type .llcrypt unregistered [ 5744.914254] Key type ._llcrypt unregistered [ 5770.931528] Key type ._llcrypt registered [ 5770.937344] Key type .llcrypt registered [ 5771.084938] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5771.511506] Key type .llcrypt unregistered [ 5771.514482] Key type ._llcrypt unregistered [ 5791.526680] Key type ._llcrypt registered [ 5791.530504] Key type .llcrypt registered [ 5791.687811] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5793.047995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5793.164514] alg: No test for adler32 (adler32-zlib) [ 5794.201471] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 5794.458417] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5796.160426] Key type lgssc registered [ 5797.651125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5807.300073] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5807.337656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5823.014703] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5828.128303] LustreError: 173443:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 5854.880915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5862.767864] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5869.908975] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5877.894154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.590182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5878.672245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5883.589492] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5883.874491] LustreError: 173468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.906387] LustreError: 173468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5888.998185] LustreError: 174368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.020890] LustreError: 174368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5892.562638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5892.974090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5898.068907] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5904.468882] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5909.431967] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5912.353330] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5923.811283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5923.827908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5923.831222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5923.839756] Lustre: Skipped 1 previous similar message [ 5928.954980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5928.958487] Lustre: Skipped 1 previous similar message [ 5929.909919] Lustre: server umount lustre-OST0000 complete [ 5939.171790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5939.176561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5939.177281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5939.177289] Lustre: Skipped 1 previous similar message [ 5939.198115] Lustre: Skipped 1 previous similar message [ 5941.342165] Lustre: server umount lustre-MDT0000 complete [ 5944.289480] LustreError: 174896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5944.301902] LustreError: 174896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5945.550614] LustreError: 173451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698860 with bad export cookie 1106873333297609061 [ 5945.559436] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5945.575197] LustreError: 173451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5945.981676] Lustre: server umount lustre-MDT0001 complete [ 5959.372625] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5963.393255] Key type lgssc unregistered [ 5963.845945] LNet: 177051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5963.882371] LNetError: 177051:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5963.907885] LNet: Removed LNI 192.168.204.115@tcp [ 5965.178096] Key type .llcrypt unregistered [ 5965.180829] Key type ._llcrypt unregistered [ 5994.745743] Key type ._llcrypt registered [ 5994.749533] Key type .llcrypt registered [ 5994.904875] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5996.513320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5996.604838] alg: No test for adler32 (adler32-zlib) [ 5997.836491] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 5998.201689] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5999.952368] Key type lgssc registered [ 6001.543763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6011.010537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6011.035578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6012.713448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6017.876693] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6022.128122] LustreError: 178450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6022.167466] LustreError: 178450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6027.235705] LustreError: 178449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6027.994065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6028.452034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6033.360636] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6039.490733] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6043.992596] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6052.369063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6052.953709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6057.110083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6061.939166] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6069.476665] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6074.977194] Lustre: Failing over lustre-MDT0000 [ 6075.328576] Lustre: server umount lustre-MDT0000 complete [ 6076.596736] LustreError: 178444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6077.925675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6077.942539] Lustre: Skipped 2 previous similar messages [ 6081.684712] LustreError: 178446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6081.699678] LustreError: 178446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6086.804339] LustreError: 178444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.828198] LustreError: 178444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6093.226642] Lustre: 177761:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698992/real 1779698992] req@00000000e487732f x1866149567425536/t0(0) o400->MGC192.168.204.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779699008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6093.263858] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6096.360153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6097.052991] LustreError: 178444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6097.081083] LustreError: 178444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 6103.532136] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xaabde85e09ccd905 [ 6103.550975] Lustre: MGC192.168.204.115@tcp: Connection restored to 0@lo (at 0@lo) [ 6103.855722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6103.933268] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6108.301501] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6108.989830] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6109.167157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6109.220643] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6109.299597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6114.339951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6120.907560] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6134.759260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6134.778541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6134.798368] Lustre: Skipped 1 previous similar message [ 6134.829363] Lustre: Skipped 1 previous similar message [ 6139.041582] Lustre: server umount lustre-OST0000 complete [ 6145.009001] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6145.020604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6145.025037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6145.084026] Lustre: Skipped 1 previous similar message [ 6150.113496] LustreError: 178444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6150.129776] LustreError: 178444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6150.214371] Lustre: server umount lustre-MDT0000 complete [ 6155.684053] LustreError: 181675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699070 with bad export cookie 12303245247635052805 [ 6155.692433] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6155.715059] LustreError: 181675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6156.101913] Lustre: server umount lustre-MDT0001 complete [ 6170.176974] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6173.679213] Key type lgssc unregistered [ 6173.946298] LNet: 182931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6173.954320] LNetError: 182931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6173.985030] LNet: Removed LNI 192.168.204.115@tcp [ 6174.772327] Key type .llcrypt unregistered [ 6174.774588] Key type ._llcrypt unregistered [ 6202.834666] Key type ._llcrypt registered [ 6202.837216] Key type .llcrypt registered [ 6202.974413] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6204.232151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6204.384173] alg: No test for adler32 (adler32-zlib) [ 6205.590877] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 6205.879125] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6207.616191] Key type lgssc registered [ 6208.699938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6217.237889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6217.257388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6218.931410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6223.723340] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6228.451273] LustreError: 184325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.465307] LustreError: 184325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6232.676341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6233.298285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6237.756351] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6244.938700] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6250.306911] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6258.733188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6259.322066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6263.479527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6267.554457] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6274.364328] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6278.481916] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6286.414803] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6298.090179] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6314.977469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6314.978968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6314.984708] LustreError: Skipped 1 previous similar message [ 6314.988520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6315.001724] Lustre: Skipped 1 previous similar message [ 6315.021493] Lustre: Skipped 1 previous similar message [ 6317.128960] Lustre: server umount lustre-OST0000 complete [ 6323.169771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6323.191970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6323.202762] Lustre: Skipped 1 previous similar message [ 6327.909859] Lustre: server umount lustre-MDT0000 complete [ 6328.290479] LustreError: 184325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6328.299843] LustreError: 184325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6332.263267] LustreError: 185044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699247 with bad export cookie 13107937786728793030 [ 6332.268357] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6332.272903] LustreError: 185044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6332.658842] Lustre: server umount lustre-MDT0001 complete [ 6344.252309] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6347.382436] Key type lgssc unregistered [ 6347.728743] LNet: 187706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6347.751592] LNetError: 187706:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6347.774220] LNet: Removed LNI 192.168.204.115@tcp [ 6348.770145] Key type .llcrypt unregistered [ 6348.771828] Key type ._llcrypt unregistered [ 6376.982803] Key type ._llcrypt registered [ 6376.985785] Key type .llcrypt registered [ 6377.115632] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6378.361055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6378.728355] alg: No test for adler32 (adler32-zlib) [ 6379.807891] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 6380.139196] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6381.840175] Key type lgssc registered [ 6383.301571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6392.254967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6392.290904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6393.703352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6400.058156] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6403.046369] LustreError: 189103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6403.074126] LustreError: 189103:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6408.178045] LustreError: 189102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6410.186947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6410.729731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6415.737132] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6421.585975] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6425.788413] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6432.667530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6433.048969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6438.201893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6440.383374] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6446.919087] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6453.472398] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6466.141974] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6473.365906] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6478.706861] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6489.572335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.576257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6489.602681] Lustre: Skipped 1 previous similar message [ 6494.690702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6494.703317] Lustre: Skipped 2 previous similar messages [ 6499.812121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6499.820723] Lustre: Skipped 1 previous similar message [ 6502.368546] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6502.641338] Lustre: server umount lustre-OST0000 complete [ 6508.004498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6508.010763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6508.020455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6510.053797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6515.169884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6515.179462] Lustre: Skipped 3 previous similar messages [ 6521.312387] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6521.638829] Lustre: server umount lustre-MDT0000 complete [ 6525.413901] LustreError: 189102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.446758] LustreError: 189102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6525.858969] LustreError: 189084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699440 with bad export cookie 15952657333395198525 [ 6525.866575] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6525.881437] LustreError: 189084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6526.362711] Lustre: server umount lustre-MDT0001 complete [ 6540.054586] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6543.785579] Key type lgssc unregistered [ 6544.141295] LNet: 193529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6544.150240] LNetError: 193529:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6545.213410] LNet: Removed LNI 192.168.204.115@tcp [ 6546.058144] Key type .llcrypt unregistered [ 6546.063921] Key type ._llcrypt unregistered [ 6573.135360] Key type ._llcrypt registered [ 6573.144262] Key type .llcrypt registered [ 6573.316206] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6574.713870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6574.820090] alg: No test for adler32 (adler32-zlib) [ 6575.996280] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 6576.299420] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6578.032182] Key type lgssc registered [ 6579.419331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6588.200971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6588.238811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6589.748990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6594.458661] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6599.142898] LustreError: 194923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6599.167799] LustreError: 194923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6603.706719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6604.126186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6609.199530] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6615.409196] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6619.694615] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6628.136442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6628.513946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6634.144880] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6634.680856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6639.265293] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6655.460923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6655.477659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6655.492617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6655.507302] LustreError: Skipped 1 previous similar message [ 6660.023567] Lustre: server umount lustre-OST0000 complete [ 6665.699182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6665.705414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6665.720209] Lustre: Skipped 2 previous similar messages [ 6665.736399] Lustre: Skipped 2 previous similar messages [ 6670.199090] Lustre: server umount lustre-MDT0000 complete [ 6670.819132] LustreError: 194924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6670.833466] LustreError: 194924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6673.886183] LustreError: 194903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699588 with bad export cookie 16729008108478315204 [ 6673.888387] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6673.900677] LustreError: 194903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6674.245792] Lustre: server umount lustre-MDT0001 complete [ 6685.380934] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6688.362667] Key type lgssc unregistered [ 6688.743304] LNet: 198251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6688.756501] LNetError: 198251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6689.836740] LNet: Removed LNI 192.168.204.115@tcp [ 6690.654558] Key type .llcrypt unregistered [ 6690.658078] Key type ._llcrypt unregistered [ 6710.183683] Key type ._llcrypt registered [ 6710.186983] Key type .llcrypt registered [ 6710.337401] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6711.731403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6711.941420] alg: No test for adler32 (adler32-zlib) [ 6713.275433] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 6713.659905] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6715.408503] Key type lgssc registered [ 6716.762359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6725.749164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6725.780613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.416421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6732.386334] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6737.909281] LustreError: 199503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6737.940993] LustreError: 199503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6741.495335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.952777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6747.379581] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6753.595456] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6758.413750] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6766.611335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6767.017416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.155423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6773.816880] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6781.230444] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6801.915618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6801.925118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6801.961028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6806.397656] Lustre: server umount lustre-OST0000 complete [ 6812.129783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6812.135845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6812.159955] Lustre: Skipped 2 previous similar messages [ 6812.169520] Lustre: Skipped 1 previous similar message [ 6817.119860] Lustre: server umount lustre-MDT0000 complete [ 6817.253753] LustreError: 200213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6817.278903] LustreError: 200213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6821.517345] LustreError: 200220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699736 with bad export cookie 18093832895626186471 [ 6821.523187] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6821.528737] LustreError: 200220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6821.904266] Lustre: server umount lustre-MDT0001 complete [ 6833.938700] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6837.739252] Key type lgssc unregistered [ 6838.152711] LNet: 202841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6838.170017] LNetError: 202841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6838.239638] LNet: Removed LNI 192.168.204.115@tcp [ 6839.360274] Key type .llcrypt unregistered [ 6839.362180] Key type ._llcrypt unregistered [ 6859.587426] Key type ._llcrypt registered [ 6859.589811] Key type .llcrypt registered [ 6859.748511] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6861.272749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6861.322964] alg: No test for adler32 (adler32-zlib) [ 6862.471870] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 6862.795470] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6864.424581] Key type lgssc registered [ 6865.782471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6873.669505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6873.717327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6875.186335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6879.726764] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6885.351326] LustreError: 204095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.372169] LustreError: 204095:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6888.867134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6889.346415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6894.455269] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6900.777063] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6905.558751] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6914.451886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6914.939809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6920.214614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6923.415339] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6930.752380] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6940.713646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6940.820565] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6941.177349] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6948.457263] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6955.336751] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6957.421551] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6969.562495] Lustre: Permanently deactivating lustre-OST0001 [ 6969.573242] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6978.174784] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6986.196752] Lustre: Permanently reactivating lustre-OST0001 [ 6986.217840] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6986.221899] Lustre: Skipped 2 previous similar messages [ 6995.033526] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.045480] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6995.053420] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6995.069249] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6995.128826] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6996.971352] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6996.995671] Lustre: Skipped 1 previous similar message [ 6997.001698] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6997.012636] Lustre: Skipped 1 previous similar message [ 6997.784558] Lustre: server umount lustre-OST0001 complete [ 7002.084919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7002.088366] LustreError: 205690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7002.092074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7002.092083] Lustre: Skipped 1 previous similar message [ 7002.098693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7002.153583] LustreError: 205690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7007.206148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7007.217117] Lustre: Skipped 2 previous similar messages [ 7007.217142] LustreError: 205690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7007.241508] LustreError: 205690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7008.187868] Lustre: server umount lustre-OST0000 complete [ 7017.443391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7017.466791] Lustre: Skipped 1 previous similar message [ 7017.478640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7017.485546] Lustre: Skipped 1 previous similar message [ 7018.955548] Lustre: server umount lustre-MDT0000 complete [ 7022.562469] LustreError: 204092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7022.576137] LustreError: 204092:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7022.615418] LustreError: 204075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699937 with bad export cookie 4978301190221231664 [ 7022.627775] LustreError: 204075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7022.629088] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7022.856874] Lustre: server umount lustre-MDT0001 complete [ 7034.045241] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7037.120840] Key type lgssc unregistered [ 7037.424086] LNet: 208669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7037.442305] LNetError: 208669:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7037.457143] LNet: Removed LNI 192.168.204.115@tcp [ 7038.363191] Key type .llcrypt unregistered [ 7038.368840] Key type ._llcrypt unregistered [ 7080.538296] Key type ._llcrypt registered [ 7080.545581] Key type .llcrypt registered [ 7080.738311] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7082.237169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7082.313482] alg: No test for adler32 (adler32-zlib) [ 7083.379957] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 7083.642330] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7085.312165] Key type lgssc registered [ 7086.616555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7094.351890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7094.366817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7095.731358] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7095.773271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7095.919863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7100.409601] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7109.453859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7109.512824] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7109.539590] Lustre: 210858:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7109.819477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7114.139812] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7120.569390] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7124.332571] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7131.202871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.409450] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7131.634076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7138.691688] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7139.977792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7145.564061] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7148.294444] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7157.566649] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7166.258783] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7174.253948] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7182.616813] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7209.071323] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7209.078578] Lustre: Skipped 2 previous similar messages [ 7243.572737] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7243.579340] Lustre: Skipped 4 previous similar messages [ 7316.596242] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7316.599520] Lustre: Skipped 8 previous similar messages [ 7324.131399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7324.138728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.148953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7324.645842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.668148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7329.455580] Lustre: server umount lustre-OST0000 complete [ 7334.881316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.883627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7334.888128] Lustre: Skipped 1 previous similar message [ 7334.897603] Lustre: Skipped 1 previous similar message [ 7340.001625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7340.009281] Lustre: Skipped 1 previous similar message [ 7344.035418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7344.042253] Lustre: Skipped 3 previous similar messages [ 7347.168351] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7347.350445] Lustre: server umount lustre-MDT0000 complete [ 7350.855596] LustreError: 210850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700265 with bad export cookie 9583332524804812783 [ 7350.857930] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7350.866495] LustreError: 210850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7351.159288] Lustre: server umount lustre-MDT0001 complete [ 7361.381236] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7363.888606] Key type lgssc unregistered [ 7364.158413] LNet: 215327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7364.172753] LNetError: 215327:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7364.187398] LNet: Removed LNI 192.168.204.115@tcp [ 7365.014228] Key type .llcrypt unregistered [ 7365.019037] Key type ._llcrypt unregistered [ 7390.686674] Key type ._llcrypt registered [ 7390.692476] Key type .llcrypt registered [ 7390.903388] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7392.606299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7392.707074] alg: No test for adler32 (adler32-zlib) [ 7394.090466] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 7394.527135] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7396.352205] Key type lgssc registered [ 7397.596381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7406.477746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7406.508629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7408.034763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7412.238879] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7418.341600] LustreError: 216723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.354565] LustreError: 216723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7422.174219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7422.831050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7428.848734] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7435.452415] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7439.640708] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7446.532896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7447.075856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7453.240925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7453.408267] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7459.502761] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7477.526930] Lustre: server umount lustre-OST0000 complete [ 7483.874959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7483.891355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7483.896575] Lustre: Skipped 1 previous similar message [ 7488.993993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7489.010971] Lustre: Skipped 2 previous similar messages [ 7491.167630] Lustre: server umount lustre-MDT0000 complete [ 7494.113593] Lustre: 216034:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779700393/real 1779700393] req@00000000ed4879a5 x1866151031245568/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779700409 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7494.113646] LustreError: 216719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7494.146718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7494.162493] LustreError: 216719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7495.001067] LustreError: 216704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700410 with bad export cookie 15882139895709030507 [ 7495.016274] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7495.772330] Lustre: server umount lustre-MDT0001 complete [ 7506.132608] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7509.205156] Key type lgssc unregistered [ 7509.513733] LNet: 220056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7509.521794] LNetError: 220056:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7509.547315] LNet: Removed LNI 192.168.204.115@tcp [ 7510.445245] Key type .llcrypt unregistered [ 7510.450847] Key type ._llcrypt unregistered [ 7535.688429] Key type ._llcrypt registered [ 7535.690509] Key type .llcrypt registered [ 7535.915181] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7537.562603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7537.731290] alg: No test for adler32 (adler32-zlib) [ 7539.004858] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 7539.319508] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7540.976338] Key type lgssc registered [ 7542.594957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7816.275626] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7819.297911] Key type lgssc unregistered [ 7819.660703] LNet: 222050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7819.670536] LNetError: 222050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7819.691078] LNet: Removed LNI 192.168.204.115@tcp [ 7820.596911] Key type .llcrypt unregistered [ 7820.603122] Key type ._llcrypt unregistered [ 7847.108247] Key type ._llcrypt registered [ 7847.110351] Key type .llcrypt registered [ 7847.266733] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 7861.519290] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7862.584070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7862.638467] alg: No test for adler32 (adler32-zlib) [ 7863.745842] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 7864.018086] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7865.752195] Key type lgssc registered [ 7866.808737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7895.095072] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7554 sec ========== 05:20:08 (1779700808) [ 7896.563254] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:20:10 (1779700810) === [ 7899.917693] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:20:13 (1779700813) === [ 7939.976414] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7943.257933] Key type lgssc unregistered [ 7943.617841] LNet: 227234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7943.626834] LNetError: 227234:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7943.639603] LNet: Removed LNI 192.168.204.115@tcp [ 7944.632657] Key type .llcrypt unregistered [ 7944.635604] Key type ._llcrypt unregistered