[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 462667328 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003149] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008592] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011004] pid_max: default: 32768 minimum: 301 [ 0.012125] LSM: Security Framework initializing [ 0.013049] Yama: becoming mindful. [ 0.014037] SELinux: Initializing. [ 0.015085] *** VALIDATE selinux *** [ 0.023811] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028261] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031100] *** VALIDATE tmpfs *** [ 0.032402] *** VALIDATE proc *** [ 0.034019] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.036246] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039027] Spectre V2 : User space: Vulnerable [ 0.040006] Speculative Store Bypass: Vulnerable [ 0.042904] debug: unmapping init [mem 0xffffffffbd259000-0xffffffffbd260fff] [ 0.044167] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045626] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053305] rcu: Hierarchical SRCU implementation. [ 0.055348] smp: Bringing up secondary CPUs ... [ 0.056586] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.060538] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132500] node 0 deferred pages initialised in 67ms [ 0.135257] devtmpfs: initialized [ 0.136216] x86/mm: Memory block size: 128MB [ 0.138610] gcov: version magic: 0x41383552 [ 0.139582] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142216] pinctrl core: initialized pinctrl subsystem [ 0.143157] [ 0.143600] ************************************************************* [ 0.144013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145012] ** ** [ 0.146016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147015] ** ** [ 0.148013] ** This means that this kernel is built to expose internal ** [ 0.149015] ** IOMMU data structures, which may compromise security on ** [ 0.150011] ** your system. ** [ 0.151013] ** ** [ 0.152011] ** If you see this message and you are not debugging the ** [ 0.153014] ** kernel, report this immediately to your vendor! ** [ 0.154013] ** ** [ 0.155012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156013] ************************************************************* [ 0.157745] NET: Registered protocol family 16 [ 0.158456] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.159057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.160129] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.161505] cpuidle: using governor menu [ 0.164460] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.166408] PCI: Using configuration type 1 for base access [ 0.168119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.177057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.178031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.180085] cryptd: max_cpu_qlen set to 1000 [ 0.182352] ACPI: Added _OSI(Module Device) [ 0.183012] ACPI: Added _OSI(Processor Device) [ 0.184012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.185011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.189481] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.192953] ACPI: Interpreter enabled [ 0.193061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.194016] ACPI: Using IOAPIC for interrupt routing [ 0.195112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.196396] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.205253] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.206034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.207019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.208073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.210197] acpiphp: Slot [2] registered [ 0.211133] acpiphp: Slot [5] registered [ 0.212099] acpiphp: Slot [6] registered [ 0.213118] acpiphp: Slot [7] registered [ 0.215162] acpiphp: Slot [8] registered [ 0.216098] acpiphp: Slot [9] registered [ 0.217108] acpiphp: Slot [10] registered [ 0.219121] acpiphp: Slot [3] registered [ 0.221115] acpiphp: Slot [4] registered [ 0.222096] acpiphp: Slot [11] registered [ 0.224098] acpiphp: Slot [12] registered [ 0.225053] acpiphp: Slot [13] registered [ 0.226119] acpiphp: Slot [14] registered [ 0.228096] acpiphp: Slot [15] registered [ 0.229101] acpiphp: Slot [16] registered [ 0.232125] acpiphp: Slot [17] registered [ 0.233116] acpiphp: Slot [18] registered [ 0.235099] acpiphp: Slot [19] registered [ 0.236173] acpiphp: Slot [20] registered [ 0.238100] acpiphp: Slot [21] registered [ 0.239100] acpiphp: Slot [22] registered [ 0.241092] acpiphp: Slot [23] registered [ 0.242104] acpiphp: Slot [24] registered [ 0.244158] acpiphp: Slot [25] registered [ 0.245113] acpiphp: Slot [26] registered [ 0.247108] acpiphp: Slot [27] registered [ 0.248098] acpiphp: Slot [28] registered [ 0.250126] acpiphp: Slot [29] registered [ 0.252095] acpiphp: Slot [30] registered [ 0.253091] acpiphp: Slot [31] registered [ 0.255091] PCI host bridge to bus 0000:00 [ 0.257024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.259021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.261020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.263020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.266023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.269040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.271135] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.274151] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.278328] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.288824] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.295000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.296016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.298019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.302018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.305113] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.307847] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.311056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.314011] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.320016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.334022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.339014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.344949] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.351000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.357015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.387016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.398845] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.405013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.412015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.435017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.450384] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.457013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.462017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.479022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.493963] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.501019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.508022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.525021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.540316] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.552015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.563015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.583022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.595017] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.602015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.607016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.624018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.633806] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.637329] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.639624] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.642376] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.645227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.649031] iommu: Default domain type: Passthrough [ 0.651452] SCSI subsystem initialized [ 0.652144] ACPI: bus type USB registered [ 0.654131] usbcore: registered new interface driver usbfs [ 0.656080] usbcore: registered new interface driver hub [ 0.659090] usbcore: registered new device driver usb [ 0.660176] pps_core: LinuxPPS API ver. 1 registered [ 0.662013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.666060] PTP clock support registered [ 0.668069] EDAC MC: Ver: 3.0.0 [ 0.670134] PCI: Using ACPI for IRQ routing [ 0.672405] NetLabel: Initializing [ 0.673009] NetLabel: domain hash size = 128 [ 0.674009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.675124] NetLabel: unlabeled traffic allowed by default [ 0.678157] vgaarb: loaded [ 0.679267] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.680011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.684405] clocksource: Switched to clocksource kvm-clock [ 0.801980] VFS: Disk quotas dquot_6.6.0 [ 0.803719] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.806342] *** VALIDATE ramfs *** [ 0.807586] *** VALIDATE hugetlbfs *** [ 0.809033] pnp: PnP ACPI init [ 0.811330] pnp: PnP ACPI: found 6 devices [ 0.830063] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.832981] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.834872] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.836796] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.839398] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.841845] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.844495] NET: Registered protocol family 2 [ 0.847033] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.852232] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.856071] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.862177] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.865694] TCP: Hash tables configured (established 65536 bind 65536) [ 0.868656] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.871836] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.875088] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.878231] NET: Registered protocol family 1 [ 0.880628] RPC: Registered named UNIX socket transport module. [ 0.882369] RPC: Registered udp transport module. [ 0.884455] RPC: Registered tcp transport module. [ 0.886773] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.889260] NET: Registered protocol family 44 [ 0.891036] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.893353] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.895676] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.898100] PCI: CLS 0 bytes, default 64 [ 0.900561] Unpacking initramfs... [ 3.119123] debug: unmapping init [mem 0xffff93c57cc54000-0xffff93c57ffbffff] [ 3.127442] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.130274] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.135434] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.189696] Initialise system trusted keyrings [ 5.196054] Key type blacklist registered [ 5.205762] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.236706] zbud: loaded [ 5.244833] *** VALIDATE nfs *** [ 5.250742] *** VALIDATE nfs4 *** [ 5.255045] pstore: using deflate compression [ 5.272243] Platform Keyring initialized [ 5.593434] NET: Registered protocol family 38 [ 5.597329] Key type asymmetric registered [ 5.600564] Asymmetric key parser 'x509' registered [ 5.611950] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.632590] io scheduler mq-deadline registered [ 5.635658] io scheduler kyber registered [ 5.641502] io scheduler bfq registered [ 5.648244] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.654548] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.664971] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.676973] ACPI: Power Button [PWRF] [ 5.703220] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.731786] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.769812] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.791567] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.830089] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.873899] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.933631] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.949671] Non-volatile memory driver v1.3 [ 5.952828] Linux agpgart interface v0.103 [ 6.065221] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 6.075627] vda: detected capacity change from 0 to 75100160 [ 6.114415] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.124989] vdb: detected capacity change from 0 to 1073741824 [ 6.177296] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.183798] vdc: detected capacity change from 0 to 2621440000 [ 6.216812] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.224963] vdd: detected capacity change from 0 to 2621440000 [ 6.256656] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.262019] vde: detected capacity change from 0 to 4294967296 [ 6.313593] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.318643] vdf: detected capacity change from 0 to 4294967296 [ 6.342120] libphy: Fixed MDIO Bus: probed [ 6.371133] usbcore: registered new interface driver usbserial_generic [ 6.380402] usbserial: USB Serial support registered for generic [ 6.387544] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.397680] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.402033] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.407215] mousedev: PS/2 mouse device common for all mice [ 6.412919] rtc_cmos 00:05: RTC can wake from S4 [ 6.417659] rtc_cmos 00:05: registered as rtc0 [ 6.421019] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.426358] intel_pstate: CPU model not supported [ 6.429606] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.434469] hid: raw HID events driver (C) Jiri Kosina [ 6.441792] usbcore: registered new interface driver usbhid [ 6.444725] usbhid: USB HID core driver [ 6.447891] drop_monitor: Initializing network drop monitor service [ 6.452560] Initializing XFRM netlink socket [ 6.455972] NET: Registered protocol family 10 [ 6.459918] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.470644] Segment Routing with IPv6 [ 6.480158] NET: Registered protocol family 17 [ 6.487435] mpls_gso: MPLS GSO support [ 6.491849] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.509207] RAS: Correctable Errors collector initialized. [ 6.517513] AVX version of gcm_enc/dec engaged. [ 6.523585] AES CTR mode by8 optimization enabled [ 6.708176] sched_clock: Marking stable (6708152768, 0)->(7672980989, -964828221) [ 6.722563] registered taskstats version 1 [ 6.725178] Loading compiled-in X.509 certificates [ 6.734836] zswap: loaded using pool lzo/zbud [ 6.820212] Key type big_key registered [ 6.841784] Key type encrypted registered [ 6.845969] ima: No TPM chip found, activating TPM-bypass! [ 6.855498] ima: Allocated hash algorithm: sha1 [ 6.859582] ima: No architecture policies found [ 6.865238] evm: Initialising EVM extended attributes: [ 6.872736] evm: security.selinux [ 6.874767] evm: security.ima [ 6.877170] evm: security.capability [ 6.881988] evm: HMAC attrs: 0x1 [ 6.898076] rtc_cmos 00:05: setting system clock to 2026-08-27 10:12:09 UTC (1787825529) [ 6.925247] debug: unmapping init [mem 0xffffffffbe203000-0xffffffffbe3fffff] [ 6.939152] debug: unmapping init [mem 0xffffffffbcf82000-0xffffffffbd258fff] [ 6.958205] Write protecting the kernel read-only data: 28672k [ 6.972467] debug: unmapping init [mem 0xffffffffbb603000-0xffffffffbb7fffff] [ 6.979698] debug: unmapping init [mem 0xffffffffbbf14000-0xffffffffbbffffff] [ 7.077350] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.106272] systemd[1]: Detected virtualization kvm. [ 7.113573] systemd[1]: Detected architecture x86-64. [ 7.127060] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.185132] systemd[1]: No hostname configured. [ 7.191107] systemd[1]: Set hostname to . [ 7.197154] random: systemd: uninitialized urandom read (16 bytes read) [ 7.205047] systemd[1]: Initializing machine ID from random generator. [ 7.560796] random: systemd: uninitialized urandom read (16 bytes read) [ 7.567144] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 7.587275] random: systemd: uninitialized urandom read (16 bytes read) [ 7.592296] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.608391] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ 7.636450] hrtimer: interrupt took 4636646 ns [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.200640] device-mapper: uevent: version 1.0.3 [ 9.206456] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] [ 11.461744] random: fast init done Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.588619] virtio_net virtio0 ens2: renamed from eth0 [ 11.854912] scsi host0: ata_piix [ 11.871426] scsi host1: ata_piix [ 11.879421] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.899606] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.406596] random: crng init done [ 16.410017] random: 7 urandom warning(s) missed due to ratelimiting [ 18.745438] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.226529] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.926297] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.964592] SELinux: Disabled at runtime. [ 24.080839] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.107904] systemd[1]: Detected virtualization kvm. [ 24.111492] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.860953] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.871091] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.878602] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.883984] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.895758] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.915157] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.952097] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 26.676206] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 27.390133] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.582988] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.690943] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.082493] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.146066] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 33.947672] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 34.867717] NFS: Registering the id_resolver key type [ 34.870221] Key type id_resolver registered [ 34.874610] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Login Service. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg639-server login: [ 87.999982] libcfs: loading out-of-tree module taints kernel. [ 88.069241] Key type ._llcrypt registered [ 88.070596] Key type .llcrypt registered [ 88.192906] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 104.399823] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 105.821424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.855718] alg: No test for adler32 (adler32-zlib) [ 107.490200] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 108.513962] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 110.255305] Key type lgssc registered [ 112.160049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.755141] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 169.414118] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 182.934141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 182.960378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.348982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.403261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 184.519357] Lustre: lustre-MDT0000: new disk, initializing [ 184.653462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.682965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 188.526655] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.617703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.711506] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.798741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 202.805508] Lustre: Skipped 1 previous similar message [ 202.944481] Lustre: lustre-MDT0001: new disk, initializing [ 203.019850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.046052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.057668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 207.280822] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.418966] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 219.324051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.559719] Lustre: lustre-OST0000: new disk, initializing [ 219.565315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 219.573778] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.638364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.254813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 223.268599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 223.344606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 224.841779] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.389233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.582808] Lustre: lustre-OST0001: new disk, initializing [ 238.588559] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 238.598557] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.693548] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.587828] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.295141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 245.308210] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.339824] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 256.446145] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 260.042609] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 266.059797] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.365657] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing yml_node [ 275.610883] Lustre: DEBUG MARKER: Client: 2.17.57.69 [ 278.595221] Lustre: DEBUG MARKER: MDS: 2.17.57.69 [ 282.019709] Lustre: DEBUG MARKER: OSS: 2.17.57.69 [ 284.628895] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 06:16:45 EDT 2026 [ 303.949335] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 305.767592] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 322.015868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.018212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.038873] Lustre: Skipped 1 previous similar message [ 327.142303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.152973] Lustre: Skipped 6 previous similar messages [ 332.255773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.264828] Lustre: Skipped 3 previous similar messages [ 336.351584] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 336.525175] Lustre: server umount lustre-MDT0000 complete [ 340.026632] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825862 with bad export cookie 6660793013263675161 [ 340.047411] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 340.054968] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.496143] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.497645] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.498186] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.507176] Lustre: Skipped 3 previous similar messages [ 342.549901] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 345.507226] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.528398] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 346.570470] Lustre: server umount lustre-MDT0001 complete [ 350.528400] Lustre: server umount lustre-OST0000 complete [ 354.532358] Lustre: server umount lustre-OST0001 complete [ 360.983899] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 371.076446] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 415.731106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.973537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 416.024318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.140345] Lustre: lustre-MDT0000: new disk, initializing [ 416.329433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.414879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.758515] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 433.319344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.420457] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 433.472153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 433.478092] Lustre: Skipped 1 previous similar message [ 433.607585] Lustre: lustre-MDT0001: new disk, initializing [ 433.716232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 433.753633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 433.761162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 437.886901] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 448.960456] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 454.955455] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 462.606824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.969434] Lustre: lustre-OST0000: new disk, initializing [ 462.980263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.989237] Lustre: 17049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 463.096593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.274110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 464.287724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 464.429119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 469.970463] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 479.165588] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 484.071208] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 484.304458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 489.362723] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 489.609857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 495.086825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 495.092034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.102025] LustreError: Skipped 1 previous similar message [ 495.102301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 495.102305] Lustre: Skipped 3 previous similar messages [ 495.147816] Lustre: Skipped 1 previous similar message [ 497.697693] Lustre: server umount lustre-OST0000 complete [ 505.325589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 505.327958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.329378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 505.329386] Lustre: Skipped 1 previous similar message [ 505.356885] Lustre: Skipped 1 previous similar message [ 507.576707] Lustre: server umount lustre-MDT0000 complete [ 510.432204] LustreError: 18388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.449409] LustreError: 18388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 511.185070] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826033 with bad export cookie 6660793013263676687 [ 511.199690] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.202795] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 511.509422] Lustre: server umount lustre-MDT0001 complete [ 514.633501] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 06:20:36 (1787826036) [ 521.838153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.349469] LustreError: 19314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.413431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 526.692071] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 527.842300] LustreError: 19315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.297615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.295747] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 547.782862] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 553.381282] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 560.406763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 560.561729] LustreError: 21134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.586756] LustreError: 21134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 567.011072] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 574.495832] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 581.610671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 581.621461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.646174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 581.653016] Lustre: Skipped 1 previous similar message [ 587.545405] Lustre: server umount lustre-OST0000 complete [ 592.363387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 592.369575] LustreError: Skipped 1 previous similar message [ 592.373675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.382801] Lustre: Skipped 1 previous similar message [ 598.371385] Lustre: server umount lustre-MDT0000 complete [ 602.245322] LustreError: 19295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826124 with bad export cookie 6660793013263677576 [ 602.251340] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 602.256646] LustreError: 19295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 602.648919] Lustre: server umount lustre-MDT0001 complete [ 612.925797] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 616.134678] Key type lgssc unregistered [ 616.466429] LNet: 22850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.477907] LNetError: 22850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 616.501054] LNet: Removed LNI 192.168.206.139@tcp [ 617.713517] Key type .llcrypt unregistered [ 617.715707] Key type ._llcrypt unregistered [ 644.376734] Key type ._llcrypt registered [ 644.379277] Key type .llcrypt registered [ 644.507207] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 646.135268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 646.298304] alg: No test for adler32 (adler32-zlib) [ 647.564153] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 647.831884] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 649.599297] Key type lgssc registered [ 651.125198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 659.055529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 659.077637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 660.552751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.492966] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 670.689786] LustreError: 24266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.703246] LustreError: 24266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 673.242334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.574534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 677.556569] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 683.859377] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 689.496327] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 696.490554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 696.710732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 703.377930] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 705.977157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 710.129466] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 726.519797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 726.520986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 726.522397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 726.542064] Lustre: Skipped 1 previous similar message [ 728.007320] Lustre: server umount lustre-OST0000 complete [ 735.204561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 735.217925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.246396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.261961] Lustre: Skipped 1 previous similar message [ 736.736945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.737531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 736.760485] Lustre: Skipped 1 previous similar message [ 738.208537] Lustre: server umount lustre-MDT0000 complete [ 741.857367] LustreError: 24266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.873385] LustreError: 24266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 742.235534] LustreError: 24249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826264 with bad export cookie 12605553433952672913 [ 742.238510] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.248359] LustreError: 24249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 742.528914] Lustre: server umount lustre-MDT0001 complete [ 752.641127] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 755.287055] Key type lgssc unregistered [ 755.628437] LNet: 28243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.641215] LNetError: 28243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.658861] LNet: Removed LNI 192.168.206.139@tcp [ 756.603673] Key type .llcrypt unregistered [ 756.606173] Key type ._llcrypt unregistered [ 779.188533] Key type ._llcrypt registered [ 779.191619] Key type .llcrypt registered [ 779.358312] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 780.834884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 781.164490] alg: No test for adler32 (adler32-zlib) [ 782.372685] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 782.677521] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 784.399263] Key type lgssc registered [ 785.958638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 794.513927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 794.571127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.987814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 800.857671] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 806.373745] LustreError: 29658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 806.387251] LustreError: 29658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 810.133818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.519859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 815.046327] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 822.628727] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 828.422432] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 842.949165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 843.273585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 849.480953] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 857.531887] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 866.892392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 875.999971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 876.006901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 876.023312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 877.026615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 877.044211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.056168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.980861] Lustre: server umount lustre-OST0000 complete [ 887.264407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 887.269214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.271802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 887.291737] Lustre: Skipped 1 previous similar message [ 887.301974] Lustre: Skipped 2 previous similar messages [ 891.480360] Lustre: server umount lustre-MDT0000 complete [ 892.385282] LustreError: 30386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 892.415787] LustreError: 30386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 895.033989] LustreError: 29638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826417 with bad export cookie 11202296965124881116 [ 895.038276] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 895.044135] LustreError: 29638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 895.495888] Lustre: server umount lustre-MDT0001 complete [ 906.243904] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 909.379471] Key type lgssc unregistered [ 909.631965] LNet: 33563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.640143] LNetError: 33563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.661847] LNet: Removed LNI 192.168.206.139@tcp [ 910.705670] Key type .llcrypt unregistered [ 910.716572] Key type ._llcrypt unregistered [ 934.998836] Key type ._llcrypt registered [ 935.002860] Key type .llcrypt registered [ 935.118654] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 936.363085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.492735] alg: No test for adler32 (adler32-zlib) [ 937.606289] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 937.915893] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 939.591238] Key type lgssc registered [ 940.942085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 949.655871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 949.680395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.216097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.756861] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 960.492800] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.533091] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 964.664457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 965.155148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 969.388333] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 976.953236] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 982.895651] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 989.818354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 990.353764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 995.507657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 997.115704] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1004.549408] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1016.804194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1016.810474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.819422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.385179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1020.390085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.398699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.426157] Lustre: Skipped 1 previous similar message [ 1025.509545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1025.522171] Lustre: Skipped 1 previous similar message [ 1030.634459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1031.136242] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1031.410813] Lustre: server umount lustre-OST0000 complete [ 1035.752266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.780316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1035.787357] Lustre: Skipped 2 previous similar messages [ 1045.985709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1045.993428] Lustre: Skipped 3 previous similar messages [ 1050.079359] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1050.450234] Lustre: server umount lustre-MDT0000 complete [ 1051.106466] LustreError: 34975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1051.126090] LustreError: 34975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1054.272821] LustreError: 34959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826576 with bad export cookie 1265202936902132793 [ 1054.292761] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1054.302722] LustreError: 34959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1054.847718] Lustre: server umount lustre-MDT0001 complete [ 1066.595579] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 1069.941634] Key type lgssc unregistered [ 1070.281147] LNet: 38530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.284798] LNetError: 38530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1070.316271] LNet: Removed LNI 192.168.206.139@tcp [ 1071.446072] Key type .llcrypt unregistered [ 1071.447343] Key type ._llcrypt unregistered [ 1094.750788] Key type ._llcrypt registered [ 1094.753679] Key type .llcrypt registered [ 1094.879074] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 1096.189865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1096.229178] alg: No test for adler32 (adler32-zlib) [ 1097.325262] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1097.607990] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 1099.287574] Key type lgssc registered [ 1100.569919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1108.336989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1108.357777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.967484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.451361] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1120.224067] LustreError: 39946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.250086] LustreError: 39946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1125.362745] LustreError: 39945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.740048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.299228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1130.967818] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1138.169746] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1145.223448] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1153.252988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1153.642568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.682498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1159.716679] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1167.292073] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1180.278039] Lustre: lustre-OST0000: Not available for connect from 192.168.206.39@tcp (stopping) [ 1180.642859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1180.650170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.667981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1181.676245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.681279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.709557] Lustre: Skipped 2 previous similar messages [ 1185.376960] Lustre: lustre-OST0000: Not available for connect from 192.168.206.39@tcp (stopping) [ 1190.498760] Lustre: lustre-OST0000: Not available for connect from 192.168.206.39@tcp (stopping) [ 1190.514135] Lustre: Skipped 2 previous similar messages [ 1191.392039] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1191.590622] Lustre: server umount lustre-OST0000 complete [ 1211.617348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.631257] Lustre: Skipped 1 previous similar message [ 1211.638587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1211.649872] Lustre: Skipped 1 previous similar message [ 1216.855462] Lustre: server umount lustre-MDT0000 complete [ 1217.007161] LustreError: 39941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1217.031566] LustreError: 39941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1220.156600] LustreError: 39926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826742 with bad export cookie 7666877873248502879 [ 1220.164449] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1220.175973] LustreError: 39926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1226.511076] Lustre: server umount lustre-MDT0001 complete [ 1236.613873] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 1239.100828] Key type lgssc unregistered [ 1239.379659] LNet: 43657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1239.391497] LNetError: 43657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.410263] LNet: Removed LNI 192.168.206.139@tcp [ 1240.211590] Key type .llcrypt unregistered [ 1240.214212] Key type ._llcrypt unregistered [ 1261.723804] Key type ._llcrypt registered [ 1261.730189] Key type .llcrypt registered [ 1261.921759] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 1263.167657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1263.236797] alg: No test for adler32 (adler32-zlib) [ 1264.349338] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1264.585658] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 1266.247721] Key type lgssc registered [ 1267.457844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1275.113499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1275.139306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.528501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1280.365264] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1287.139166] LustreError: 45068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.151199] LustreError: 45068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1288.590760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.904810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1293.596089] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1301.320350] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1306.595589] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1312.836251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1313.168037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.912785] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1322.430536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1325.825957] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1332.340283] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.39@tcp (stopping) [ 1332.711321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1332.732889] Lustre: Skipped 2 previous similar messages [ 1333.490333] Lustre: server umount lustre-MDT0000 complete [ 1336.713534] LustreError: 45049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826859 with bad export cookie 1764680597703381942 [ 1336.720279] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1336.725173] LustreError: 45049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1337.450467] LustreError: 45062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.476673] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.39@tcp (stopping) [ 1337.484146] Lustre: Skipped 3 previous similar messages [ 1337.832242] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.857298] LustreError: 45065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.857698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1341.872153] LustreError: 45065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1341.892178] Lustre: Skipped 1 previous similar message [ 1346.978625] LustreError: 45063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.979747] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1347.001510] LustreError: 45063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1347.015502] Lustre: Skipped 1 previous similar message [ 1351.135676] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1351.545292] Lustre: server umount lustre-MDT0001 complete [ 1367.574032] Lustre: server umount lustre-OST0000 complete [ 1384.746446] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 1387.541065] Key type lgssc unregistered [ 1387.876501] LNet: 48933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.888192] LNetError: 48933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.946194] LNet: Removed LNI 192.168.206.139@tcp [ 1388.696240] Key type .llcrypt unregistered [ 1388.702226] Key type ._llcrypt unregistered [ 1411.353534] Key type ._llcrypt registered [ 1411.356137] Key type .llcrypt registered [ 1411.439453] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 1412.498935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1412.520558] alg: No test for adler32 (adler32-zlib) [ 1413.608190] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1413.824776] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 1415.519293] Key type lgssc registered [ 1416.623898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1423.787802] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1423.806673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1439.391912] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1444.511678] LustreError: 50321:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.139@tcp: failed processing log, type 4: rc = -110 [ 1471.200301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.503607] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1485.071376] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1650.945908] Lustre: server umount lustre-OST0000 complete [ 1667.925155] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 1670.645402] Key type lgssc unregistered [ 1671.099083] LNet: 52002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1671.105272] LNetError: 52002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.168302] LNet: Removed LNI 192.168.206.139@tcp [ 1673.313881] Key type .llcrypt unregistered [ 1673.317492] Key type ._llcrypt unregistered [ 1695.371249] Key type ._llcrypt registered [ 1695.374965] Key type .llcrypt registered [ 1695.485220] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 1696.533036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1696.656337] alg: No test for adler32 (adler32-zlib) [ 1697.747506] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1698.066296] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 1699.775318] Key type lgssc registered [ 1700.816213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1707.672977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1707.703650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.184510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1713.318630] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1718.758825] LustreError: 53395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1718.768837] LustreError: 53395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1720.477575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.826650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1724.955118] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1732.265548] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1738.318078] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1746.336587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1746.965240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.206680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1755.170646] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1762.812218] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1767.142555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1767.143253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.245814] Lustre: server umount lustre-OST0000 complete [ 1777.122443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1777.134948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.167203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1777.181082] Lustre: Skipped 1 previous similar message [ 1781.523074] Lustre: server umount lustre-MDT0000 complete [ 1782.753536] LustreError: 53395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1782.763824] LustreError: 53395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1785.655704] LustreError: 53377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827308 with bad export cookie 950775526837379727 [ 1785.668058] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1786.103516] Lustre: server umount lustre-MDT0001 complete [ 1797.075262] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 1800.637176] Key type lgssc unregistered [ 1801.102903] LNet: 56929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.116185] LNetError: 56929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.145814] LNet: Removed LNI 192.168.206.139@tcp [ 1802.137636] Key type .llcrypt unregistered [ 1802.139704] Key type ._llcrypt unregistered [ 1824.899975] Key type ._llcrypt registered [ 1824.902543] Key type .llcrypt registered [ 1825.055877] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 1826.094260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1826.169348] alg: No test for adler32 (adler32-zlib) [ 1827.176699] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1827.371073] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 1829.079158] Key type lgssc registered [ 1830.303142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1837.845087] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1837.872328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1853.472681] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1858.598708] LustreError: 58317:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.139@tcp: failed processing log, type 4: rc = -110 [ 1885.279606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.872823] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1899.431508] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1907.656607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.467757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.559617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1913.454243] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1913.837459] LustreError: 58344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.854316] LustreError: 58344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1918.958344] LustreError: 58343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.972540] LustreError: 58344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1920.002765] LustreError: 58344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1922.289874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.650248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1927.002587] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1934.007239] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1939.140567] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1943.011068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.015526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1943.028226] Lustre: Skipped 1 previous similar message [ 1943.033460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1943.034218] Lustre: Skipped 1 previous similar message [ 1947.274323] Lustre: server umount lustre-OST0000 complete [ 1967.585458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.592029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1967.599497] Lustre: Skipped 1 previous similar message [ 1967.608619] Lustre: Skipped 1 previous similar message [ 1972.704619] LustreError: 59358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.758298] Lustre: server umount lustre-MDT0000 complete [ 1976.360446] LustreError: 58323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827498 with bad export cookie 18436116530079806046 [ 1976.377405] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1976.377554] LustreError: 58323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1982.789692] Lustre: server umount lustre-MDT0001 complete [ 1993.120982] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 1995.825814] Key type lgssc unregistered [ 1996.145868] LNet: 62063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1996.155435] LNetError: 62063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1996.173712] LNet: Removed LNI 192.168.206.139@tcp [ 1997.110874] Key type .llcrypt unregistered [ 1997.119809] Key type ._llcrypt unregistered [ 2022.189376] Key type ._llcrypt registered [ 2022.191967] Key type .llcrypt registered [ 2022.302808] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 2023.724442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2023.754497] alg: No test for adler32 (adler32-zlib) [ 2024.912905] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2025.236940] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 2026.887181] Key type lgssc registered [ 2027.990348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2035.712511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2035.738978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.155472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.776882] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2046.439149] LustreError: 63477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.455974] LustreError: 63477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2049.471772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2049.958092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2054.526368] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2062.741566] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2068.189175] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2075.553685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2076.124188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.323892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2082.951982] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2090.704892] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2109.925677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2109.929157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.940344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2109.969032] Lustre: Skipped 1 previous similar message [ 2115.046764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2115.058635] Lustre: Skipped 1 previous similar message [ 2115.365254] Lustre: server umount lustre-OST0000 complete [ 2120.160961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.180487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2120.188895] Lustre: Skipped 2 previous similar messages [ 2125.280830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2125.294179] Lustre: Skipped 1 previous similar message [ 2125.886840] Lustre: server umount lustre-MDT0000 complete [ 2129.474945] LustreError: 66278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827652 with bad export cookie 5402593963958032281 [ 2129.475940] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2129.481782] LustreError: 66278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2129.803910] Lustre: server umount lustre-MDT0001 complete [ 2140.626386] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 2143.054922] Key type lgssc unregistered [ 2143.302532] LNet: 67035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.312698] LNetError: 67035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2143.327210] LNet: Removed LNI 192.168.206.139@tcp [ 2144.242387] Key type .llcrypt unregistered [ 2144.245725] Key type ._llcrypt unregistered [ 2169.215952] Key type ._llcrypt registered [ 2169.218520] Key type .llcrypt registered [ 2169.349502] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 2170.599400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2170.717685] alg: No test for adler32 (adler32-zlib) [ 2171.997185] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2172.401801] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 2174.111235] Key type lgssc registered [ 2175.477315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2185.089683] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 06:48:26 (1787827706) [ 2191.874388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2191.904366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.396764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2197.512911] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2202.597946] LustreError: 68740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.619873] LustreError: 68740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2205.047205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.279249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2210.156664] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2217.878885] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2224.197972] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2230.939828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2231.313107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.302996] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2237.491769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2245.999540] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2254.734867] Lustre: Failing over lustre-MDT0000 [ 2255.032141] Lustre: server umount lustre-MDT0000 complete [ 2255.840615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.849493] LustreError: 70376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.869064] Lustre: Skipped 2 previous similar messages [ 2255.930031] LustreError: 70376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2258.090720] LustreError: 68735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.961724] LustreError: 68741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2261.000807] LustreError: 68741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2266.082676] LustreError: 68737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.107957] LustreError: 68737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2266.499712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.775914] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2266.909942] Lustre: *** cfs_fail_loc=135, val=0*** [ 2266.979350] LustreError: 71827:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2266.988784] LustreError: 71827:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.139@tcp: cfg command failed: rc = -2 [ 2267.011135] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2267.011135] [ 2267.038041] LustreError: MGC192.168.206.139@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2267.066345] LustreError: 71815:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2267.080785] LustreError: 71815:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2267.094902] LustreError: 71815:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2267.121880] LustreError: 71815:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2267.158524] Lustre: server umount lustre-MDT0000 complete [ 2267.160931] LustreError: 71815:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2273.949994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2274.064474] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2274.068670] LustreError: 72172:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.206.139@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000fd3520b4) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2274.076032] LustreError: 68733:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2274.241655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2274.275550] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2274.436175] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2279.329458] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2279.416180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2279.454017] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2279.575846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2289.635018] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.642338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2289.651686] Lustre: Skipped 1 previous similar message [ 2294.755680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2294.783163] Lustre: Skipped 2 previous similar messages [ 2295.813862] Lustre: server umount lustre-OST0000 complete [ 2299.872494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2299.876373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.884810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2299.893121] Lustre: Skipped 1 previous similar message [ 2299.907614] Lustre: Skipped 1 previous similar message [ 2305.008529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2305.927991] Lustre: server umount lustre-MDT0000 complete [ 2308.067171] LustreError: 68740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.083957] LustreError: 68740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2309.595784] LustreError: 68721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827832 with bad export cookie 13908374851208526206 [ 2309.600221] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2309.604540] LustreError: 68721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2309.953829] Lustre: server umount lustre-MDT0001 complete [ 2320.314435] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 2323.489466] Key type lgssc unregistered [ 2323.782492] LNet: 73611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.791053] LNetError: 73611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.806207] LNet: Removed LNI 192.168.206.139@tcp [ 2324.768261] Key type .llcrypt unregistered [ 2324.772219] Key type ._llcrypt unregistered [ 2352.073571] Key type ._llcrypt registered [ 2352.076705] Key type .llcrypt registered [ 2352.182665] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 2353.633252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2353.666143] alg: No test for adler32 (adler32-zlib) [ 2354.733140] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2354.952065] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 2356.623664] Key type lgssc registered [ 2358.027624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2366.239791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2366.271400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.791956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2372.126535] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2378.215873] LustreError: 75025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.245326] LustreError: 75025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2380.693966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.209171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2385.677870] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2393.250596] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2398.480470] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2404.522207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2404.827346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.297091] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2413.041711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2417.451896] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2422.619886] Lustre: Failing over lustre-MDT0000 [ 2422.948099] Lustre: server umount lustre-MDT0000 complete [ 2423.272462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.282543] Lustre: Skipped 2 previous similar messages [ 2423.286704] LustreError: 76040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.299099] LustreError: 76040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2424.434885] LustreError: 75753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.391465] LustreError: 75025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.411278] LustreError: 75025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2431.184097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.356527] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2431.538153] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2431.685804] LustreError: 78111:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2431.699044] LustreError: 78111:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.139@tcp: cfg command failed: rc = -9 [ 2431.709242] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2431.709242] [ 2431.717449] LustreError: MGC192.168.206.139@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2431.739026] LustreError: 78099:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2431.749742] LustreError: 78099:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2431.756274] LustreError: 78099:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2431.763660] LustreError: 78099:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2431.785758] Lustre: server umount lustre-MDT0000 complete [ 2431.789228] LustreError: 78099:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2433.515558] LustreError: 75026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.555437] LustreError: 75026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2438.103317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.251049] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2438.523892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2438.588668] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2439.776434] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2443.328168] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2443.780459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2443.814554] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2443.876652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2451.952046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2451.958579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.975578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2453.987745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2453.990885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.688462] Lustre: server umount lustre-OST0000 complete [ 2464.224148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2464.237021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.239537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2464.263355] Lustre: Skipped 1 previous similar message [ 2464.277985] Lustre: Skipped 2 previous similar messages [ 2467.621834] Lustre: server umount lustre-MDT0000 complete [ 2469.349822] LustreError: 75020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.379138] LustreError: 75020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2470.967373] LustreError: 75005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827993 with bad export cookie 889235020593172515 [ 2470.976076] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2470.980780] LustreError: 75005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2471.288143] Lustre: server umount lustre-MDT0001 complete [ 2480.908674] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 2483.818728] Key type lgssc unregistered [ 2484.129042] LNet: 79893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.136780] LNetError: 79893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2484.154564] LNet: Removed LNI 192.168.206.139@tcp [ 2484.915382] Key type .llcrypt unregistered [ 2484.917232] Key type ._llcrypt unregistered [ 2496.537193] Key type ._llcrypt registered [ 2496.541901] Key type .llcrypt registered [ 2496.698764] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 2497.021485] Key type .llcrypt unregistered [ 2497.024902] Key type ._llcrypt unregistered [ 2518.457521] Key type ._llcrypt registered [ 2518.460887] Key type .llcrypt registered [ 2518.571885] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 2519.768035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2519.794052] alg: No test for adler32 (adler32-zlib) [ 2520.885204] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2521.168716] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 2522.911194] Key type lgssc registered [ 2523.941632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2531.728415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2531.763859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.304945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.149299] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2542.565474] LustreError: 81799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.585272] LustreError: 81799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2546.096699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.443639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2551.212996] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2558.029929] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2563.215890] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2569.648461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2569.970901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.863220] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2578.165730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2583.096545] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2593.767756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.778704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2593.787663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.814900] Lustre: Skipped 1 previous similar message [ 2598.904533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2598.915750] Lustre: Skipped 1 previous similar message [ 2599.216441] Lustre: server umount lustre-OST0000 complete [ 2604.007504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.026032] Lustre: Skipped 1 previous similar message [ 2604.035283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2604.041414] Lustre: Skipped 1 previous similar message [ 2609.121364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.129839] Lustre: Skipped 1 previous similar message [ 2609.633158] Lustre: server umount lustre-MDT0000 complete [ 2613.185735] LustreError: 81780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828135 with bad export cookie 3403626894525499471 [ 2613.190609] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2613.196789] LustreError: 81780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2613.502763] Lustre: server umount lustre-MDT0001 complete [ 2625.065631] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 2628.049799] Key type lgssc unregistered [ 2628.353109] LNet: 85336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2628.365167] LNetError: 85336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2628.411214] LNet: Removed LNI 192.168.206.139@tcp [ 2629.657723] Key type .llcrypt unregistered [ 2629.659514] Key type ._llcrypt unregistered [ 2652.612413] Key type ._llcrypt registered [ 2652.615486] Key type .llcrypt registered [ 2652.721324] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 2653.991470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2654.002861] alg: No test for adler32 (adler32-zlib) [ 2655.004528] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2655.259554] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 2656.937373] Key type lgssc registered [ 2658.024791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2664.547262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2664.570145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.917974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2669.601970] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2676.195074] LustreError: 86750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.215489] LustreError: 86750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2677.595308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.890493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2682.038532] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2689.653731] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2695.146467] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2702.579233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2702.830238] LustreError: 88570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.967178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.078223] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2710.535117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2716.380327] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2723.812095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2723.822338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2723.839409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2724.294518] Lustre: server umount lustre-OST0000 complete [ 2728.932756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2728.938182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.947791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2730.978401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2730.981299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.099065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2736.106646] Lustre: Skipped 2 previous similar messages [ 2738.276790] Lustre: server umount lustre-MDT0000 complete [ 2741.217330] LustreError: 86745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.254621] LustreError: 86745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2741.646094] LustreError: 86731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828264 with bad export cookie 9979902568404932429 [ 2741.647584] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2741.653453] LustreError: 86731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2742.030704] Lustre: server umount lustre-MDT0001 complete [ 2750.894120] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 2753.307632] Key type lgssc unregistered [ 2753.566557] LNet: 90283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2753.577650] LNetError: 90283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2753.599079] LNet: Removed LNI 192.168.206.139@tcp [ 2754.292410] Key type .llcrypt unregistered [ 2754.297443] Key type ._llcrypt unregistered [ 2774.213055] Key type ._llcrypt registered [ 2774.214779] Key type .llcrypt registered [ 2774.349764] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 2775.230138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2775.304225] alg: No test for adler32 (adler32-zlib) [ 2776.337808] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2776.517263] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 2778.191326] Key type lgssc registered [ 2779.335437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2786.779442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2786.817047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2788.273819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.978940] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2797.545204] LustreError: 91681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.563739] LustreError: 91681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2798.386113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.722671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2802.621372] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2809.634917] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2815.351988] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2821.646131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2821.886659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2828.349651] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2834.237827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2835.366871] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2847.720561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2847.727569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.735722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2849.764420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2849.766557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2849.768121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.771493] Lustre: Skipped 1 previous similar message [ 2852.175927] Lustre: server umount lustre-OST0000 complete [ 2860.003662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.006283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2860.012563] Lustre: Skipped 1 previous similar message [ 2860.012905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2860.042755] Lustre: Skipped 1 previous similar message [ 2861.739527] Lustre: server umount lustre-MDT0000 complete [ 2865.014902] LustreError: 91661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828387 with bad export cookie 11147619747447589392 [ 2865.015652] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2865.023256] LustreError: 91661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2865.268692] Lustre: server umount lustre-MDT0001 complete [ 2874.286545] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 2876.942543] Key type lgssc unregistered [ 2877.219037] LNet: 95217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2877.238527] LNetError: 95217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2878.316237] LNet: Removed LNI 192.168.206.139@tcp [ 2879.071160] Key type .llcrypt unregistered [ 2879.073349] Key type ._llcrypt unregistered [ 2898.491075] Key type ._llcrypt registered [ 2898.493929] Key type .llcrypt registered [ 2898.573214] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 2899.187184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2899.202112] alg: No test for adler32 (adler32-zlib) [ 2900.282617] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2900.442492] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 2902.079161] Key type lgssc registered [ 2903.013991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2909.450905] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2909.464097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2924.959371] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2930.079799] LustreError: 96582:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.139@tcp: failed processing log, type 4: rc = -110 [ 2956.770314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.977238] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2971.112698] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2976.168251] Lustre: server umount lustre-OST0000 complete [ 2983.908437] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 07:01:45 (1787828505) [ 2989.781490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.352196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.515575] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2995.690649] LustreError: 98235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.726255] LustreError: 98235:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2999.780741] LustreError: 98236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.530290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.816677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3006.666839] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3012.945623] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3017.951512] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3025.135778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.419393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.788379] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3034.687830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3038.971209] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3053.027739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3053.044939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3053.065642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3055.079977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3055.096937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3055.124336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3056.852287] Lustre: server umount lustre-OST0000 complete [ 3064.295091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3064.304420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.318698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3064.323910] Lustre: Skipped 1 previous similar message [ 3067.261876] Lustre: server umount lustre-MDT0000 complete [ 3070.438136] LustreError: 98230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.451765] LustreError: 98230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3070.585768] LustreError: 98218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828593 with bad export cookie 401859775958643090 [ 3070.588094] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3070.600054] LustreError: 98218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3071.104486] Lustre: server umount lustre-MDT0001 complete [ 3081.422849] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 3084.007041] Key type lgssc unregistered [ 3084.278537] LNet: 101931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3084.288397] LNetError: 101931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3084.299036] LNet: Removed LNI 192.168.206.139@tcp [ 3085.241940] Key type .llcrypt unregistered [ 3085.250168] Key type ._llcrypt unregistered [ 3122.938956] Key type ._llcrypt registered [ 3122.941342] Key type .llcrypt registered [ 3123.055560] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 3123.759056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3123.795549] alg: No test for adler32 (adler32-zlib) [ 3124.829757] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3125.146767] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 3126.815219] Key type lgssc registered [ 3127.837388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3135.044468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3135.068125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.561850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.054320] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3146.720709] LustreError: 104118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.751767] LustreError: 104118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3148.887971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.288748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3153.143486] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3159.954060] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3165.291289] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3171.857175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3172.338791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3178.545689] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3181.553019] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3187.368667] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3199.968950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3199.977919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3200.006206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3202.030127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3202.030667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3202.049439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.420132] Lustre: server umount lustre-OST0000 complete [ 3210.723550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3210.738653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.757788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3210.767194] Lustre: Skipped 1 previous similar message [ 3215.137668] Lustre: server umount lustre-MDT0000 complete [ 3217.376109] LustreError: 104118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3217.401470] LustreError: 104118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3219.071853] LustreError: 104097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828741 with bad export cookie 12276064193706002561 [ 3219.076526] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3219.079879] LustreError: 104097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3219.378799] Lustre: server umount lustre-MDT0001 complete [ 3231.640152] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 3235.179035] Key type lgssc unregistered [ 3235.574760] LNet: 107651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3235.605201] LNetError: 107651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3235.641259] LNet: Removed LNI 192.168.206.139@tcp [ 3236.624946] Key type .llcrypt unregistered [ 3236.628293] Key type ._llcrypt unregistered [ 3255.102685] Key type ._llcrypt registered [ 3255.105598] Key type .llcrypt registered [ 3255.216322] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 3256.477332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3256.532463] alg: No test for adler32 (adler32-zlib) [ 3257.619852] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3257.947359] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 3259.583585] Key type lgssc registered [ 3261.090486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3268.545825] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3268.559155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3284.127369] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3289.247586] LustreError: 108895:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.139@tcp: failed processing log, type 4: rc = -110 [ 3316.000537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3323.262554] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3330.348578] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3336.578307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.933452] LustreError: MGC192.168.206.139@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3336.942243] LustreError: 109920:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3336.947238] LustreError: 109920:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3336.957275] LustreError: 109920:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3337.048864] Lustre: server umount lustre-MDT0000 complete [ 3337.051898] LustreError: 109920:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3353.709283] Lustre: server umount lustre-OST0000 complete [ 3363.504042] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 3373.100347] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 3404.702675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3404.947771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3404.992689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3405.074348] Lustre: lustre-MDT0000: new disk, initializing [ 3405.186073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.220398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3409.683311] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3421.105913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.227264] Lustre: 112709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3421.273597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3421.283814] Lustre: Skipped 1 previous similar message [ 3421.413354] Lustre: lustre-MDT0001: new disk, initializing [ 3421.551852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3421.583401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3421.597033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3426.007942] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3434.302910] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3439.759739] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3445.654124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3445.901570] Lustre: lustre-OST0000: new disk, initializing [ 3445.911446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3445.924964] Lustre: 114677:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3446.010460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3452.120767] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3454.517772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3454.532854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3454.609623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3460.495438] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3465.181896] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3465.393762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3469.606780] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3469.786311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3475.434740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3475.441826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.475136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3475.489473] Lustre: Skipped 1 previous similar message [ 3477.877592] Lustre: server umount lustre-OST0000 complete [ 3483.106682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3483.111582] LustreError: Skipped 1 previous similar message [ 3483.114341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3483.127717] Lustre: Skipped 1 previous similar message [ 3483.132740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.668885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.672300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3487.712709] Lustre: server umount lustre-MDT0000 complete [ 3491.765240] LustreError: 112703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829014 with bad export cookie 744321481649644523 [ 3491.766964] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3491.776157] LustreError: 112703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3492.236859] Lustre: server umount lustre-MDT0001 complete [ 3501.732730] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 07:10:23 (1787829023) [ 3505.024761] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3521.778923] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 3532.188990] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 3566.995386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3567.332507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3567.374721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3567.478336] Lustre: lustre-MDT0000: new disk, initializing [ 3567.544285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3567.569816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3572.472368] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3584.033413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3584.139422] Lustre: 119574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3584.219728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3584.227929] Lustre: Skipped 1 previous similar message [ 3584.306367] Lustre: lustre-MDT0001: new disk, initializing [ 3584.451746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3584.483875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3584.498315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3588.472214] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3600.080770] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3606.361660] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3614.674594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3614.993165] Lustre: lustre-OST0000: new disk, initializing [ 3614.996472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3615.005412] Lustre: 121545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3615.082805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3621.997864] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3623.575163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3623.600672] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3623.770571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3632.524806] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3637.439275] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3637.695595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3642.089720] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3642.366956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3649.514180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3649.516623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3649.541906] Lustre: Skipped 1 previous similar message [ 3649.546125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3649.556549] Lustre: Skipped 2 previous similar messages [ 3650.902320] Lustre: server umount lustre-OST0000 complete [ 3656.164378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3656.171944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3656.186819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.456455] Lustre: server umount lustre-MDT0000 complete [ 3664.868948] LustreError: 120488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3664.884443] LustreError: 120488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3665.305659] LustreError: 119569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829187 with bad export cookie 744321481649645405 [ 3665.307445] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3665.324035] LustreError: 119569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3665.878358] Lustre: server umount lustre-MDT0001 complete [ 3675.677665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3676.125353] LustreError: 123763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.201386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3681.250826] LustreError: 123764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3681.655534] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3686.369424] LustreError: 123763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.492440] LustreError: 123764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.569155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3696.796504] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3704.648538] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3710.194796] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3716.829892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3722.492990] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3729.024342] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3744.026560] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3751.923359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3751.924625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.938333] LustreError: Skipped 1 previous similar message [ 3751.944900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3751.951094] Lustre: Skipped 2 previous similar messages [ 3751.971680] Lustre: Skipped 3 previous similar messages [ 3754.397862] Lustre: server umount lustre-OST0000 complete [ 3764.546055] Lustre: server umount lustre-MDT0000 complete [ 3767.267240] LustreError: 123759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3767.290710] LustreError: 123759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3767.857540] LustreError: 123743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829290 with bad export cookie 744321481649646287 [ 3767.865130] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3767.881552] LustreError: 123743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3768.226432] Lustre: server umount lustre-MDT0001 complete [ 3778.547309] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 3781.523539] Key type lgssc unregistered [ 3781.959927] LNet: 127420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3781.990476] LNetError: 127420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3782.010484] LNet: Removed LNI 192.168.206.139@tcp [ 3782.878782] Key type .llcrypt unregistered [ 3782.883269] Key type ._llcrypt unregistered [ 3800.518873] Key type ._llcrypt registered [ 3800.522724] Key type .llcrypt registered [ 3800.648175] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 3814.684348] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 3815.603344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3815.728524] alg: No test for adler32 (adler32-zlib) [ 3816.754760] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3816.979563] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 3818.639160] Key type lgssc registered [ 3819.930899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3851.634823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3851.663395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3852.874661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3852.904342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3852.976518] Lustre: lustre-MDT0000: new disk, initializing [ 3853.045934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3853.071947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3857.201467] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3866.586921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.723693] Lustre: 130587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3866.795604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3866.801489] Lustre: Skipped 1 previous similar message [ 3866.929353] Lustre: lustre-MDT0001: new disk, initializing [ 3867.064068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3867.123943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3867.135677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3872.137636] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3881.233198] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3887.139847] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3893.738550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3894.034720] Lustre: lustre-OST0000: new disk, initializing [ 3894.038540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3894.052493] Lustre: 132556:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3894.155307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.801369] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3902.546106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3902.559606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3902.656071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3909.888922] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3915.019703] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3915.246734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3920.187166] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3920.489522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3923.451902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3923.454204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.466028] LustreError: Skipped 1 previous similar message [ 3923.468815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3923.498553] Lustre: Skipped 1 previous similar message [ 3928.548534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3928.557035] Lustre: Skipped 1 previous similar message [ 3933.675423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3933.683635] Lustre: Skipped 1 previous similar message [ 3937.247160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3937.540533] Lustre: server umount lustre-OST0000 complete [ 3943.916060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3943.928038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3943.931414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3943.931423] Lustre: Skipped 1 previous similar message [ 3943.938872] Lustre: Skipped 1 previous similar message [ 3947.925670] Lustre: server umount lustre-MDT0000 complete [ 3952.274412] LustreError: 130578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829474 with bad export cookie 11962280823318903206 [ 3952.281293] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3952.282775] LustreError: 130578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3952.783527] Lustre: server umount lustre-MDT0001 complete [ 3961.420430] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 07:18:02 (1787829482) [ 3968.954198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.576146] LustreError: 134970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3969.600881] LustreError: 134970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3969.706867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3974.991767] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3975.137457] LustreError: 134971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.257948] LustreError: 134970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3983.733748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3984.274321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3989.630238] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3998.855992] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4005.270393] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4007.732513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.734635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4007.749047] Lustre: Skipped 1 previous similar message [ 4007.764661] Lustre: Skipped 2 previous similar messages [ 4019.169373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4019.183058] Lustre: Skipped 5 previous similar messages [ 4022.240026] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4022.415884] Lustre: server umount lustre-MDT0000 complete [ 4024.289930] LustreError: 135698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4024.314486] LustreError: 135698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4027.154513] LustreError: 134950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829549 with bad export cookie 11962280823318904095 [ 4027.169085] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4027.176459] LustreError: 134950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4027.854073] Lustre: server umount lustre-MDT0001 complete [ 4036.192318] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 07:19:17 (1787829557) [ 4042.356323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4057.954138] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4063.071470] LustreError: 137386:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.139@tcp: failed processing log, type 4: rc = -110 [ 4088.799314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.996676] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4103.898556] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4106.668586] Lustre: server umount lustre-OST0000 complete [ 4116.105204] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 07:20:37 (1787829637) [ 4123.166094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4123.741991] LustreError: 138832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.866686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4128.657491] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4134.375667] LustreError: 138833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4134.397853] LustreError: 138833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4137.923439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.336886] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4151.215651] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4158.182123] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4164.975502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4165.319838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.326647] Lustre: Skipped 1 previous similar message [ 4171.205079] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4178.671528] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4194.794961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4194.820032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.851286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4200.708932] Lustre: server umount lustre-MDT0000 complete [ 4203.492776] LustreError: 138832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.531607] LustreError: 138832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4204.614788] LustreError: 138812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829727 with bad export cookie 11962280823318904760 [ 4204.622430] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4204.635160] LustreError: 138812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4204.965052] Lustre: server umount lustre-MDT0001 complete [ 4219.420248] Lustre: server umount lustre-OST0000 complete [ 4228.075496] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 07:22:29 (1787829749) [ 4237.100324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4237.628417] LustreError: 142506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.670229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.956772] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4252.682158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4258.265864] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4266.299149] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4271.506089] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4278.016506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4282.448541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4285.162324] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4294.528375] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4299.024650] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4299.271349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4304.697501] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4304.966427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4307.936934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4307.941991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.948140] Lustre: Skipped 2 previous similar messages [ 4307.951875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4307.957698] Lustre: Skipped 3 previous similar messages [ 4313.827557] Lustre: server umount lustre-OST0000 complete [ 4319.722991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4319.738140] LustreError: Skipped 1 previous similar message [ 4319.739961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4319.757691] Lustre: Skipped 1 previous similar message [ 4325.036587] Lustre: server umount lustre-MDT0000 complete [ 4328.940043] LustreError: 142507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4328.978064] LustreError: 142507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4329.059676] LustreError: 142489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829851 with bad export cookie 11962280823318905831 [ 4329.063119] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4329.071128] LustreError: 142489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4329.465512] Lustre: server umount lustre-MDT0001 complete [ 4338.720782] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 07:24:19 (1787829859) [ 4345.808504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4361.311739] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4366.431531] LustreError: 146666:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.139@tcp: failed processing log, type 4: rc = -110 [ 4392.159382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.166763] Lustre: Skipped 2 previous similar messages [ 4398.073500] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4405.744219] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4411.786501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4412.213191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4416.152060] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4425.884730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4431.664654] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4438.551981] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4443.969845] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4448.978257] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4449.210918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4453.369610] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4453.535931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4456.931404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4456.940279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4456.963474] Lustre: Skipped 1 previous similar message [ 4456.969023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4456.979037] Lustre: Skipped 7 previous similar messages [ 4461.767995] Lustre: server umount lustre-MDT0000 complete [ 4463.073554] LustreError: 147704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.095840] LustreError: 147704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4465.437191] LustreError: 149815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829988 with bad export cookie 11962280823318906853 [ 4465.453664] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4465.460000] LustreError: 149815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4465.813532] Lustre: server umount lustre-MDT0001 complete [ 4481.166318] Lustre: server umount lustre-OST0000 complete [ 4490.556445] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 07:26:51 (1787830011) [ 4497.302561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4512.928623] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4518.051287] LustreError: 150862:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.139@tcp: failed processing log, type 4: rc = -110 [ 4549.262499] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4555.963586] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4562.728570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.335535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4567.307653] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4575.521239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4580.113922] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4588.293531] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4594.312196] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4601.025783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4601.247114] Lustre: lustre-OST0001: new disk, initializing [ 4601.252518] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4601.264907] Lustre: 153736:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4607.614859] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4609.122613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4609.138561] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4609.288466] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4617.540580] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4622.881682] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4623.104469] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4629.260789] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4629.513859] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4632.559114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4632.569904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.592321] Lustre: Skipped 2 previous similar messages [ 4632.620041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4632.629502] Lustre: Skipped 3 previous similar messages [ 4637.690766] Lustre: server umount lustre-OST0000 complete [ 4648.352899] Lustre: server umount lustre-OST0001 complete [ 4667.359166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4668.120904] Lustre: server umount lustre-MDT0000 complete [ 4674.084076] LustreError: 150870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830196 with bad export cookie 11962280823318907805 [ 4674.089514] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4674.121694] LustreError: 150870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4674.767441] Lustre: server umount lustre-MDT0001 complete [ 4704.464878] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 07:30:25 (1787830225) [ 4706.298639] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4708.058436] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4709.606815] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 07:30:31 (1787830231) [ 4716.415958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4716.780508] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4716.851104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4717.172418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4717.180234] Lustre: Skipped 6 previous similar messages [ 4721.736906] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4730.342773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4730.431919] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4730.451389] Lustre: 157296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4734.612994] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4741.852672] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4747.492969] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4753.502956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4753.760790] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4760.166140] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4760.234548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4768.309755] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4774.112969] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4774.387633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4779.206620] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4779.405426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4786.147300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4786.150808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4786.161040] Lustre: Skipped 5 previous similar messages [ 4786.174088] LustreError: Skipped 4 previous similar messages [ 4787.785650] Lustre: server umount lustre-OST0000 complete [ 4809.508187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4816.376707] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4824.909975] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4826.235819] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4826.264319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4826.268380] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4826.284040] Lustre: Skipped 1 previous similar message [ 4830.152527] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4830.377905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4835.663734] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4835.883952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4841.602397] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f4645a1c000.ost_server_uuid 50 [ 4843.194349] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f4645a1c000.ost_server_uuid in FULL state after 0 sec [ 4843.381283] LustreError: 157303:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.39@tcp) returned error from blocking AST (req@00000000c4dbd819 x1874674937243008 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000046de9bf8/0xa6028f07a96414f7 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.39@tcp remote: 0x7e1cb51fed57a0d5 expref: 5 pid: 157305 timeout: 4943 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4843.419881] LustreError: lustre-MDT0000: A client on nid 192.168.206.39@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4843.444740] LustreError: 157293:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.39@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000046de9bf8/0xa6028f07a96414f7 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.39@tcp remote: 0x7e1cb51fed57a0d5 expref: 6 pid: 157305 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4852.193641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4852.208855] LustreError: Skipped 1 previous similar message [ 4854.113304] Lustre: server umount lustre-OST0000 complete [ 4872.159326] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4872.320334] Lustre: server umount lustre-MDT0000 complete [ 4876.623175] LustreError: 162668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830399 with bad export cookie 11962280823318908988 [ 4876.623744] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4876.633794] LustreError: 162668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4883.328942] Lustre: server umount lustre-MDT0001 complete [ 4896.963745] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 4900.814728] Key type lgssc unregistered [ 4901.216848] LNet: 163426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4901.223302] LNetError: 163426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4901.246378] LNet: Removed LNI 192.168.206.139@tcp [ 4902.248111] Key type .llcrypt unregistered [ 4902.251880] Key type ._llcrypt unregistered [ 4927.282047] Key type ._llcrypt registered [ 4927.284318] Key type .llcrypt registered [ 4927.459576] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 4929.521196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4929.581119] alg: No test for adler32 (adler32-zlib) [ 4930.739933] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 4931.100855] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 4932.791328] Key type lgssc registered [ 4934.210677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4944.292458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4944.306042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4945.982354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4951.581193] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4956.138618] LustreError: 164841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.152413] LustreError: 164841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4960.493632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4960.869523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4965.265468] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4971.608815] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4976.998198] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4983.381186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4983.720246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.339773] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4990.933916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4996.654802] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5001.728920] Lustre: Failing over lustre-MDT0000 [ 5001.914886] Lustre: server umount lustre-MDT0000 complete [ 5003.419354] LustreError: 164835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.314795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5006.315859] LustreError: 164836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.337904] Lustre: Skipped 2 previous similar messages [ 5006.362379] LustreError: 164836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5011.425992] LustreError: 164837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5011.449067] LustreError: 164837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5013.936220] LustreError: 164820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.39@tcp arrived at 1787830536 with bad export cookie 3289814935404542749 [ 5013.971714] LustreError: 164820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5016.554230] LustreError: 167388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5016.591083] LustreError: 167388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5021.664490] Lustre: 164151:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787830528/real 1787830528] req@00000000c536418c x1874676105110016/t0(0) o400->MGC192.168.206.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787830544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5021.687839] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5022.452387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.790448] LustreError: 165569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.812154] LustreError: 165569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5031.936800] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2da7c540e98ffb7f [ 5031.950139] Lustre: MGC192.168.206.139@tcp: Connection restored to 0@lo (at 0@lo) [ 5032.321130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5032.422307] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5037.207652] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5037.540224] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5037.558401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5039.364207] Lustre: lustre-MDT0000: Denying connection for new client 19430776-2fbb-482a-bfc7-f0d62e2de5ae (at 192.168.206.39@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5047.780451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5047.801340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5052.385649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5052.392751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.406772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5053.674979] Lustre: server umount lustre-OST0000 complete [ 5056.925345] LustreError: 168574:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5056.931431] Lustre: 167923:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5056.941539] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5056.956130] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5056.966786] LustreError: 167923:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d2a1b4db x1874676105129088/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5056.982598] LustreError: 167923:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5056.991962] LustreError: 167923:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5057.000921] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5057.394810] Lustre: server umount lustre-MDT0000 complete [ 5058.016856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5058.028088] LustreError: 167388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5058.042233] LustreError: 167388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5060.794078] LustreError: 164821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830583 with bad export cookie 3289814935404542847 [ 5060.799036] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5060.810760] LustreError: 164821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5067.126606] Lustre: server umount lustre-MDT0001 complete [ 5076.868564] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 5079.334820] Key type lgssc unregistered [ 5079.568119] LNet: 169329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5079.577380] LNetError: 169329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5079.603401] LNet: Removed LNI 192.168.206.139@tcp [ 5080.369100] Key type .llcrypt unregistered [ 5080.371451] Key type ._llcrypt unregistered [ 5101.201736] Key type ._llcrypt registered [ 5101.203706] Key type .llcrypt registered [ 5101.276579] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 5102.169306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5102.190120] alg: No test for adler32 (adler32-zlib) [ 5103.236426] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 5103.462350] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 5105.159243] Key type lgssc registered [ 5106.251453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5114.635662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5114.687986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5116.070329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5120.251580] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5125.601354] LustreError: 170722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5125.637474] LustreError: 170722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5129.240889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5129.765444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5134.074798] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5141.069412] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5146.396870] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5152.566099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5153.115127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5159.583851] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5161.370682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5166.586743] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5175.594687] Lustre: server umount lustre-OST0000 complete [ 5180.896637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5180.904818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5180.925960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5181.928154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5181.932878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5187.041747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5187.052913] Lustre: Skipped 2 previous similar messages [ 5190.181542] Lustre: server umount lustre-MDT0000 complete [ 5191.647106] Lustre: 170033:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787830698/real 1787830698] req@00000000b052f91d x1874676286515456/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787830714 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5191.664027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.672612] LustreError: 170719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.696681] LustreError: 170719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5193.183630] Lustre: 170032:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787830699/real 1787830699] req@00000000ba487318 x1874676286516352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787830715 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5193.411451] LustreError: 170702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830716 with bad export cookie 13774961309836560595 [ 5193.413521] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5193.431593] LustreError: 170702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5193.899516] Lustre: server umount lustre-MDT0001 complete [ 5207.251147] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 5210.713560] Key type lgssc unregistered [ 5211.063382] LNet: 174257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5211.070872] LNetError: 174257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5211.084428] LNet: Removed LNI 192.168.206.139@tcp [ 5212.390441] Key type .llcrypt unregistered [ 5212.392764] Key type ._llcrypt unregistered [ 5248.868340] Key type ._llcrypt registered [ 5248.871345] Key type .llcrypt registered [ 5248.968553] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 5250.105094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5250.141654] alg: No test for adler32 (adler32-zlib) [ 5251.173935] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 5251.371622] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 5253.088803] Key type lgssc registered [ 5254.126522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5260.841468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5260.851380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5262.394523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5266.696595] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5271.527874] LustreError: 175964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5271.550359] LustreError: 175964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5275.310468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5275.830446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5280.509527] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5288.471413] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5294.908563] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5302.855486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5303.338329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5306.429204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5309.721887] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5317.162533] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5330.913327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5330.920580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5330.922604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5335.888652] Lustre: server umount lustre-OST0000 complete [ 5341.152331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5341.156746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5341.181538] Lustre: Skipped 2 previous similar messages [ 5341.186195] Lustre: Skipped 1 previous similar message [ 5345.801982] Lustre: server umount lustre-MDT0000 complete [ 5346.272347] LustreError: 177204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5346.293969] LustreError: 177204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5349.644167] LustreError: 175946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830872 with bad export cookie 13149862622378018571 [ 5349.650863] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5349.660731] LustreError: 175946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5350.134631] Lustre: server umount lustre-MDT0001 complete [ 5360.676840] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 5364.827912] Key type lgssc unregistered [ 5365.199032] LNet: 179524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5365.215587] LNetError: 179524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5365.239159] LNet: Removed LNI 192.168.206.139@tcp [ 5366.272160] Key type .llcrypt unregistered [ 5366.273951] Key type ._llcrypt unregistered [ 5390.256953] Key type ._llcrypt registered [ 5390.265349] Key type .llcrypt registered [ 5390.406435] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 5391.678209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5391.718619] alg: No test for adler32 (adler32-zlib) [ 5392.813111] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 5393.079430] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 5394.743297] Key type lgssc registered [ 5395.740790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5405.867228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5405.900218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.285396] Lustre: *** cfs_fail_loc=135, val=0*** [ 5407.402774] LustreError: 181047:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5407.416862] LustreError: 181047:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.139@tcp: cfg command failed: rc = -2 [ 5407.425604] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5407.425604] [ 5407.431591] LustreError: MGC192.168.206.139@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5407.439120] LustreError: 181008:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5407.443580] LustreError: 181008:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5407.448586] LustreError: 181008:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5407.454158] LustreError: 181008:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5407.523236] Lustre: server umount lustre-MDT0000 complete [ 5407.538053] LustreError: 181008:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5414.305636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5414.676073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5418.425697] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5425.126845] LustreError: 181418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5425.163637] LustreError: 181418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5426.312776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5426.772223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5430.800190] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5437.897593] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5443.487527] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5447.139688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5447.152655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5447.173415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5450.209743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5450.210386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.257433] LustreError: 181413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.268476] LustreError: 181413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5453.516802] Lustre: server umount lustre-MDT0000 complete [ 5457.024896] LustreError: 181398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830979 with bad export cookie 11352180291098221755 [ 5457.026049] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5457.038276] LustreError: 181398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5457.515595] Lustre: server umount lustre-MDT0001 complete [ 5468.917523] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 5471.831712] Key type lgssc unregistered [ 5472.111697] LNet: 183741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5472.124088] LNetError: 183741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5472.146195] LNet: Removed LNI 192.168.206.139@tcp [ 5473.048560] Key type .llcrypt unregistered [ 5473.050256] Key type ._llcrypt unregistered [ 5497.477215] Key type ._llcrypt registered [ 5497.481826] Key type .llcrypt registered [ 5497.654035] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 5498.145219] Key type .llcrypt unregistered [ 5498.148174] Key type ._llcrypt unregistered [ 5514.814202] Key type ._llcrypt registered [ 5514.818630] Key type .llcrypt registered [ 5514.919892] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 5515.906158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5516.041632] alg: No test for adler32 (adler32-zlib) [ 5517.059645] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 5517.327543] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 5519.015212] Key type lgssc registered [ 5520.253720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5528.355714] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5528.373312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5543.904843] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5549.023719] LustreError: 185880:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.139@tcp: failed processing log, type 4: rc = -110 [ 5575.651016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5582.683679] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5591.159486] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5598.295963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.929290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5598.973669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5603.419711] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5604.329795] LustreError: 185906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5604.350875] LustreError: 185906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5609.451480] LustreError: 185905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5610.475705] LustreError: 186918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5610.496487] LustreError: 186918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5611.378694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.733554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5615.850434] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5622.715234] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5627.575267] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5630.749542] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5642.730518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.731326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5642.736389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5642.746774] Lustre: Skipped 1 previous similar message [ 5647.845173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5647.860660] Lustre: Skipped 2 previous similar messages [ 5648.692492] Lustre: server umount lustre-OST0000 complete [ 5652.970378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5652.973615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.982070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5653.004112] Lustre: Skipped 1 previous similar message [ 5653.020645] Lustre: Skipped 1 previous similar message [ 5658.080626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5658.086813] Lustre: Skipped 1 previous similar message [ 5658.765355] Lustre: server umount lustre-MDT0000 complete [ 5662.540758] LustreError: 185886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831185 with bad export cookie 16829829840165986367 [ 5662.556641] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5662.563926] LustreError: 185886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5663.081372] Lustre: server umount lustre-MDT0001 complete [ 5672.953376] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 5675.616346] Key type lgssc unregistered [ 5675.883610] LNet: 189896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5675.890696] LNetError: 189896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5675.905402] LNet: Removed LNI 192.168.206.139@tcp [ 5676.718247] Key type .llcrypt unregistered [ 5676.722437] Key type ._llcrypt unregistered [ 5697.097629] Key type ._llcrypt registered [ 5697.102486] Key type .llcrypt registered [ 5697.207653] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 5698.080598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5698.097919] alg: No test for adler32 (adler32-zlib) [ 5699.138847] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 5699.358783] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 5701.063217] Key type lgssc registered [ 5701.944521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5708.149331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5708.166134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.567366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.612722] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5720.038443] LustreError: 191288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5720.048147] LustreError: 191288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5721.100623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5721.491178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5725.807931] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5732.472165] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5739.082935] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5746.458306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5746.875701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5752.494945] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5755.059607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5759.801114] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5764.694817] Lustre: Failing over lustre-MDT0000 [ 5764.996767] Lustre: server umount lustre-MDT0000 complete [ 5765.602378] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.605717] LustreError: 191289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5765.621082] Lustre: Skipped 1 previous similar message [ 5765.640471] LustreError: 191289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5766.630807] LustreError: 192324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.753956] LustreError: 191289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.770065] LustreError: 191289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5776.867196] LustreError: 192324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.883706] LustreError: 192324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5781.990230] Lustre: 190598:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831288/real 1787831288] req@00000000ede56b12 x1874676910414976/t0(0) o400->MGC192.168.206.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787831304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5782.006744] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5782.725314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.107854] LustreError: 191288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.126037] LustreError: 191288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5792.364353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.401194] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5792.866130] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5796.105904] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5797.865431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5797.889074] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5797.939116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5800.387067] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5807.855648] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5824.992621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5825.008323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5825.008617] LustreError: Skipped 1 previous similar message [ 5825.010348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5825.074071] Lustre: Skipped 2 previous similar messages [ 5828.584698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5828.597099] Lustre: Skipped 2 previous similar messages [ 5830.610296] Lustre: server umount lustre-OST0000 complete [ 5838.818057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5838.846147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5838.851976] Lustre: Skipped 1 previous similar message [ 5840.659198] Lustre: server umount lustre-MDT0000 complete [ 5843.940909] LustreError: 192324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.961795] LustreError: 192324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5845.001288] LustreError: 191268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831367 with bad export cookie 9907934644796791961 [ 5845.006470] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5845.011473] LustreError: 191268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5845.403130] Lustre: server umount lustre-MDT0001 complete [ 5856.741043] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 5860.029679] Key type lgssc unregistered [ 5860.441475] LNet: 196201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5860.451654] LNetError: 196201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5860.470837] LNet: Removed LNI 192.168.206.139@tcp [ 5861.509388] Key type .llcrypt unregistered [ 5861.511949] Key type ._llcrypt unregistered [ 5885.417732] Key type ._llcrypt registered [ 5885.420270] Key type .llcrypt registered [ 5885.505640] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 5886.453406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5886.473221] alg: No test for adler32 (adler32-zlib) [ 5887.503407] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 5887.693860] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 5889.343221] Key type lgssc registered [ 5890.286232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5898.148867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5898.162032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5899.535213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5903.584669] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5909.988352] LustreError: 197614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5909.996981] LustreError: 197614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5910.361823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5910.600171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5914.008108] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5920.484242] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5924.581536] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5929.816718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5930.158534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5934.486243] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 5940.430513] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5944.489227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5948.015984] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5950.317375] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5961.679894] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5970.421292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5970.424669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5970.435868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5970.447727] Lustre: Skipped 1 previous similar message [ 5970.461600] Lustre: Skipped 1 previous similar message [ 5975.003797] Lustre: server umount lustre-OST0000 complete [ 5980.645438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5980.660822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5980.663242] Lustre: Skipped 1 previous similar message [ 5980.675825] Lustre: Skipped 1 previous similar message [ 5984.717223] Lustre: server umount lustre-MDT0000 complete [ 5985.766093] LustreError: 197611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5985.788327] LustreError: 197611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5988.241489] LustreError: 197596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831510 with bad export cookie 9761707831834008362 [ 5988.244185] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5988.254033] LustreError: 197596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5988.703216] Lustre: server umount lustre-MDT0001 complete [ 5999.229445] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 6001.817408] Key type lgssc unregistered [ 6002.145265] LNet: 201464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6002.151329] LNetError: 201464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6002.177661] LNet: Removed LNI 192.168.206.139@tcp [ 6003.035939] Key type .llcrypt unregistered [ 6003.038299] Key type ._llcrypt unregistered [ 6018.863615] Key type ._llcrypt registered [ 6018.865962] Key type .llcrypt registered [ 6018.932184] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 6019.916093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6020.086322] alg: No test for adler32 (adler32-zlib) [ 6021.079946] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 6021.259108] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 6022.943263] Key type lgssc registered [ 6023.865733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6030.160474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6030.172084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6031.544163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6035.414584] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6042.084600] LustreError: 202709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6042.108588] LustreError: 202709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6043.860465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6044.289582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6048.444577] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6054.718311] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6060.135207] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6066.330536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6066.727501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6072.514716] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6076.990359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6079.363880] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6084.948972] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6097.999375] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6104.054668] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6109.637936] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6122.978188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6122.981692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.988389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6122.997433] Lustre: Skipped 1 previous similar message [ 6126.864980] Lustre: server umount lustre-OST0000 complete [ 6131.168581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6131.175902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6131.203780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6131.209409] Lustre: Skipped 1 previous similar message [ 6133.216865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6133.217613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6135.957515] Lustre: server umount lustre-MDT0000 complete [ 6138.338232] LustreError: 202710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6138.352511] LustreError: 202710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6138.948063] LustreError: 205388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831661 with bad export cookie 10885688635167248439 [ 6138.949616] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6138.961494] LustreError: 205388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6139.207894] Lustre: server umount lustre-MDT0001 complete [ 6148.663172] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 6151.154268] Key type lgssc unregistered [ 6151.431342] LNet: 207634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6151.438818] LNetError: 207634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6152.503562] LNet: Removed LNI 192.168.206.139@tcp [ 6153.246231] Key type .llcrypt unregistered [ 6153.251844] Key type ._llcrypt unregistered [ 6173.368858] Key type ._llcrypt registered [ 6173.379909] Key type .llcrypt registered [ 6173.473527] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 6174.443765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6174.472042] alg: No test for adler32 (adler32-zlib) [ 6175.477435] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 6175.684364] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 6177.359150] Key type lgssc registered [ 6178.238694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6184.485623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6184.499795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6185.919126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6189.624619] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6196.205430] LustreError: 209028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6196.244283] LustreError: 209028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6196.410845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6196.929861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6201.095418] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6207.936427] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6212.502863] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6218.120241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6218.461897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6223.944987] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6230.777930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6231.209313] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6246.368126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6246.379169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.381185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6246.392891] Lustre: Skipped 1 previous similar message [ 6251.382333] Lustre: server umount lustre-OST0000 complete [ 6256.611154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6256.615096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6256.634030] Lustre: Skipped 1 previous similar message [ 6256.646378] Lustre: Skipped 2 previous similar messages [ 6261.249989] Lustre: server umount lustre-MDT0000 complete [ 6261.749356] LustreError: 209029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6261.776542] LustreError: 209029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6264.917156] LustreError: 209763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831787 with bad export cookie 10563155052586905528 [ 6264.926357] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6264.929728] LustreError: 209763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6265.202357] Lustre: server umount lustre-MDT0001 complete [ 6274.871739] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 6277.610515] Key type lgssc unregistered [ 6277.842488] LNet: 212711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6277.850968] LNetError: 212711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6277.867944] LNet: Removed LNI 192.168.206.139@tcp [ 6278.777880] Key type .llcrypt unregistered [ 6278.783118] Key type ._llcrypt unregistered [ 6297.965693] Key type ._llcrypt registered [ 6297.969401] Key type .llcrypt registered [ 6298.090440] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 6299.176135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6299.196565] alg: No test for adler32 (adler32-zlib) [ 6300.203969] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 6300.407507] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 6302.063474] Key type lgssc registered [ 6302.998673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6310.543815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6310.598397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6312.428904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6317.367582] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6321.647697] LustreError: 213978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6321.670534] LustreError: 213978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6326.394967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6326.810529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6331.947940] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6339.692771] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6345.272715] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6352.033539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6352.313228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6357.434609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6358.266856] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6365.138262] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6383.073678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6383.083343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6383.083738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6383.089784] Lustre: Skipped 1 previous similar message [ 6385.701565] Lustre: server umount lustre-OST0000 complete [ 6393.314973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6393.315970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.330886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6393.354739] Lustre: Skipped 1 previous similar message [ 6393.371488] Lustre: Skipped 2 previous similar messages [ 6396.460372] Lustre: server umount lustre-MDT0000 complete [ 6398.431957] LustreError: 213978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6398.450614] LustreError: 213978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6400.207840] LustreError: 214712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831922 with bad export cookie 18057153846634097810 [ 6400.209642] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6400.222056] LustreError: 214712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6400.650896] Lustre: server umount lustre-MDT0001 complete [ 6411.653426] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 6413.909342] Key type lgssc unregistered [ 6414.145269] LNet: 217662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6414.157106] LNetError: 217662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6414.171525] LNet: Removed LNI 192.168.206.139@tcp [ 6414.775935] Key type .llcrypt unregistered [ 6414.779149] Key type ._llcrypt unregistered [ 6429.684920] Key type ._llcrypt registered [ 6429.690156] Key type .llcrypt registered [ 6429.784428] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 6430.341942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6430.396386] alg: No test for adler32 (adler32-zlib) [ 6431.356748] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 6431.586873] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 6433.231165] Key type lgssc registered [ 6434.251645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6441.145079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6441.162201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6442.466610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6446.543235] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6452.705083] LustreError: 218906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6452.726102] LustreError: 218906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6453.921578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.203945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6457.688764] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6463.948315] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6468.404995] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6474.168584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6474.550481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6480.405712] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6487.858492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6488.159853] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6495.766302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6495.842821] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6496.076528] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6501.150146] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6508.058247] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6509.526326] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6520.985952] Lustre: Permanently deactivating lustre-OST0001 [ 6521.000718] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6528.567060] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6534.290473] Lustre: Permanently reactivating lustre-OST0001 [ 6534.331680] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6534.337975] Lustre: Skipped 2 previous similar messages [ 6535.308853] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.320730] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6535.328678] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6535.337917] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6535.384270] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6542.306738] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6542.326526] Lustre: Skipped 2 previous similar messages [ 6542.329583] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6542.337960] Lustre: Skipped 1 previous similar message [ 6544.447182] Lustre: server umount lustre-OST0001 complete [ 6547.429748] LustreError: 220726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6547.452699] LustreError: 220726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6549.474079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6549.484531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6549.495525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6550.504194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6550.512387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6552.549293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6552.551604] LustreError: 220728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6552.556022] Lustre: Skipped 1 previous similar message [ 6552.582624] LustreError: 220728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6553.663364] Lustre: server umount lustre-OST0000 complete [ 6557.671680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6557.674340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6557.694196] Lustre: Skipped 2 previous similar messages [ 6563.255810] Lustre: server umount lustre-MDT0000 complete [ 6566.134504] LustreError: 218889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832088 with bad export cookie 12119751329952622234 [ 6566.136405] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6566.145179] LustreError: 218889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6566.524759] Lustre: server umount lustre-MDT0001 complete [ 6574.784920] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 6577.547889] Key type lgssc unregistered [ 6577.880593] LNet: 223969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6577.887648] LNetError: 223969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6577.902980] LNet: Removed LNI 192.168.206.139@tcp [ 6578.927587] Key type .llcrypt unregistered [ 6578.930629] Key type ._llcrypt unregistered [ 6613.214461] Key type ._llcrypt registered [ 6613.218964] Key type .llcrypt registered [ 6613.312557] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 6614.159660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6614.220575] alg: No test for adler32 (adler32-zlib) [ 6615.310489] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 6615.596714] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 6617.255758] Key type lgssc registered [ 6618.337772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6625.084820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6625.106594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6626.365061] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6626.410819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6626.608071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6630.853858] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6639.220730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.300213] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6639.357303] Lustre: 226150:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6639.658080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6643.727684] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6650.644580] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6655.795172] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6661.763733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.947121] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6662.099876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6667.561916] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6668.402662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6674.427921] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6677.116330] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6683.620564] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6693.359702] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6699.937796] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6708.484618] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6728.626489] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6728.635157] Lustre: Skipped 2 previous similar messages [ 6764.346561] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6764.359902] Lustre: Skipped 4 previous similar messages [ 6835.524686] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6835.529467] Lustre: Skipped 8 previous similar messages [ 6842.859446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.863025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6842.884254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6847.457367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6847.469514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6847.484826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6848.929114] Lustre: server umount lustre-OST0000 complete [ 6853.093819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6853.115167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6853.122290] Lustre: Skipped 2 previous similar messages [ 6858.210488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6858.229120] Lustre: Skipped 1 previous similar message [ 6859.538454] Lustre: server umount lustre-MDT0000 complete [ 6863.841608] LustreError: 226144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832386 with bad export cookie 5400257632228594461 [ 6863.869179] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6864.705330] Lustre: server umount lustre-MDT0001 complete [ 6877.738157] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 6881.054863] Key type lgssc unregistered [ 6881.529618] LNet: 230965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6881.548578] LNetError: 230965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6881.576928] LNet: Removed LNI 192.168.206.139@tcp [ 6882.505144] Key type .llcrypt unregistered [ 6882.507753] Key type ._llcrypt unregistered [ 6907.775945] Key type ._llcrypt registered [ 6907.777854] Key type .llcrypt registered [ 6907.925077] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 6909.257495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6909.290532] alg: No test for adler32 (adler32-zlib) [ 6910.286918] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 6910.541667] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 6912.192951] Key type lgssc registered [ 6913.479721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6922.097535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6922.125824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.773833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6929.105825] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6933.987660] LustreError: 232378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6933.999041] LustreError: 232378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6938.585317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6939.075530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6944.452543] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6954.639797] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6960.393113] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6966.520057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6966.939239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6969.035628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6972.800603] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6980.315353] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6994.918809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.921812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6994.924779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6994.942846] Lustre: Skipped 1 previous similar message [ 7000.066384] Lustre: server umount lustre-OST0000 complete [ 7005.153641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7005.172622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7005.179609] Lustre: Skipped 2 previous similar messages [ 7010.280689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7010.845612] Lustre: server umount lustre-MDT0000 complete [ 7015.392176] LustreError: 233105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.416568] LustreError: 233105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7015.744587] LustreError: 234958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832538 with bad export cookie 9521081612862376397 [ 7015.746917] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7015.759580] LustreError: 234958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7016.000047] Lustre: server umount lustre-MDT0001 complete [ 7028.181311] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 7031.105846] Key type lgssc unregistered [ 7031.380163] LNet: 236068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7031.386423] LNetError: 236068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7031.404763] LNet: Removed LNI 192.168.206.139@tcp [ 7032.640533] Key type .llcrypt unregistered [ 7032.643345] Key type ._llcrypt unregistered [ 7059.591747] Key type ._llcrypt registered [ 7059.593725] Key type .llcrypt registered [ 7059.684862] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 7060.538637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7060.552334] alg: No test for adler32 (adler32-zlib) [ 7061.602554] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 7061.835773] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 7063.495220] Key type lgssc registered [ 7064.742764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7335.337677] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 7338.044304] Key type lgssc unregistered [ 7338.344778] LNet: 238100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7338.358226] LNetError: 238100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7338.380397] LNet: Removed LNI 192.168.206.139@tcp [ 7339.263520] Key type .llcrypt unregistered [ 7339.269299] Key type ._llcrypt unregistered [ 7364.169213] Key type ._llcrypt registered [ 7364.180314] Key type .llcrypt registered [ 7364.384507] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 7380.369809] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 7381.385587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7381.533609] alg: No test for adler32 (adler32-zlib) [ 7382.613074] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 7382.805804] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 7384.488358] Key type lgssc registered [ 7385.450865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7413.506811] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7128 sec ========== 08:15:35 (1787832935) [ 7414.946637] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:15:36 (1787832936) === [ 7417.913657] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:15:39 (1787832939) === [ 7453.191322] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 7455.474130] Key type lgssc unregistered [ 7455.703766] LNet: 243335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7455.719332] LNetError: 243335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7455.736042] LNet: Removed LNI 192.168.206.139@tcp [ 7456.347136] Key type .llcrypt unregistered [ 7456.351859] Key type ._llcrypt unregistered