[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487889969 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003273] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008009] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011121] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014057] *** VALIDATE selinux *** [ 0.022614] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027311] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] *** VALIDATE tmpfs *** [ 0.031420] *** VALIDATE proc *** [ 0.032247] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034247] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.041199] debug: unmapping init [mem 0xffffffff9b259000-0xffffffff9b260fff] [ 0.043255] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044754] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045023] ... version: 2 [ 0.046014] ... bit width: 48 [ 0.047017] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052267] rcu: Hierarchical SRCU implementation. [ 0.054495] smp: Bringing up secondary CPUs ... [ 0.055679] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.059572] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.169299] node 0 deferred pages initialised in 104ms [ 0.173021] devtmpfs: initialized [ 0.174265] x86/mm: Memory block size: 128MB [ 0.177873] gcov: version magic: 0x41383552 [ 0.180160] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.181081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.183309] pinctrl core: initialized pinctrl subsystem [ 0.185183] [ 0.185714] ************************************************************* [ 0.187013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189014] ** ** [ 0.192016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.194009] ** ** [ 0.196012] ** This means that this kernel is built to expose internal ** [ 0.198013] ** IOMMU data structures, which may compromise security on ** [ 0.200014] ** your system. ** [ 0.202013] ** ** [ 0.205015] ** If you see this message and you are not debugging the ** [ 0.208014] ** kernel, report this immediately to your vendor! ** [ 0.209010] ** ** [ 0.211012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.212012] ************************************************************* [ 0.214801] NET: Registered protocol family 16 [ 0.216282] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.219066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.222049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.225114] cpuidle: using governor menu [ 0.226572] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.228508] PCI: Using configuration type 1 for base access [ 0.231145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.242150] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.243053] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.245080] cryptd: max_cpu_qlen set to 1000 [ 0.248192] ACPI: Added _OSI(Module Device) [ 0.250030] ACPI: Added _OSI(Processor Device) [ 0.251035] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.253022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.259000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.267629] ACPI: Interpreter enabled [ 0.268089] ACPI: PM: (supports S0 S3 S4 S5) [ 0.269019] ACPI: Using IOAPIC for interrupt routing [ 0.271082] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.275375] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.286933] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.290045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.293018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.296101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.303380] acpiphp: Slot [2] registered [ 0.305339] acpiphp: Slot [5] registered [ 0.307192] acpiphp: Slot [6] registered [ 0.308174] acpiphp: Slot [7] registered [ 0.310253] acpiphp: Slot [8] registered [ 0.312277] acpiphp: Slot [9] registered [ 0.313155] acpiphp: Slot [10] registered [ 0.315107] acpiphp: Slot [3] registered [ 0.316169] acpiphp: Slot [4] registered [ 0.317100] acpiphp: Slot [11] registered [ 0.319119] acpiphp: Slot [12] registered [ 0.320113] acpiphp: Slot [13] registered [ 0.322109] acpiphp: Slot [14] registered [ 0.323122] acpiphp: Slot [15] registered [ 0.325111] acpiphp: Slot [16] registered [ 0.327109] acpiphp: Slot [17] registered [ 0.328111] acpiphp: Slot [18] registered [ 0.329102] acpiphp: Slot [19] registered [ 0.331128] acpiphp: Slot [20] registered [ 0.332105] acpiphp: Slot [21] registered [ 0.334096] acpiphp: Slot [22] registered [ 0.336127] acpiphp: Slot [23] registered [ 0.337097] acpiphp: Slot [24] registered [ 0.338283] acpiphp: Slot [25] registered [ 0.340121] acpiphp: Slot [26] registered [ 0.342268] acpiphp: Slot [27] registered [ 0.343111] acpiphp: Slot [28] registered [ 0.345163] acpiphp: Slot [29] registered [ 0.346114] acpiphp: Slot [30] registered [ 0.348177] acpiphp: Slot [31] registered [ 0.349079] PCI host bridge to bus 0000:00 [ 0.351030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.354037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.356031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.359036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.362042] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.365038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.368192] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.371075] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.375489] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.386778] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.391997] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.394028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.397060] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.401029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.404691] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.407760] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.410046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.412912] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.419018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.433019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.437025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.442000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.451027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.458023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.481025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.494196] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.504024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.512015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.538028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.548000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.557014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.566016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.585030] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.597958] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.611022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.617022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.640023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.652781] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.666025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.674022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.694016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.710687] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.719025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.726025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.745000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.754000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.756312] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.759022] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.760415] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.763281] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.768150] iommu: Default domain type: Passthrough [ 0.773486] SCSI subsystem initialized [ 0.775244] ACPI: bus type USB registered [ 0.777137] usbcore: registered new interface driver usbfs [ 0.779099] usbcore: registered new interface driver hub [ 0.781098] usbcore: registered new device driver usb [ 0.782330] pps_core: LinuxPPS API ver. 1 registered [ 0.784013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.787093] PTP clock support registered [ 0.789156] EDAC MC: Ver: 3.0.0 [ 0.791140] PCI: Using ACPI for IRQ routing [ 0.793122] NetLabel: Initializing [ 0.794014] NetLabel: domain hash size = 128 [ 0.796016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.798093] NetLabel: unlabeled traffic allowed by default [ 0.800112] vgaarb: loaded [ 0.802393] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.804024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.811000] clocksource: Switched to clocksource kvm-clock [ 0.926639] VFS: Disk quotas dquot_6.6.0 [ 0.928446] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.931686] *** VALIDATE ramfs *** [ 0.933257] *** VALIDATE hugetlbfs *** [ 0.935224] pnp: PnP ACPI init [ 0.938536] pnp: PnP ACPI: found 6 devices [ 0.958905] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.962494] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.965142] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.967732] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.970530] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.973398] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.976606] NET: Registered protocol family 2 [ 0.979454] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.984754] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.988495] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.993838] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.997179] TCP: Hash tables configured (established 65536 bind 65536) [ 1.000061] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.003022] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.006258] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.009457] NET: Registered protocol family 1 [ 1.012385] RPC: Registered named UNIX socket transport module. [ 1.014749] RPC: Registered udp transport module. [ 1.018297] RPC: Registered tcp transport module. [ 1.021203] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.025178] NET: Registered protocol family 44 [ 1.027895] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.031328] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.034498] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.037981] PCI: CLS 0 bytes, default 64 [ 1.041300] Unpacking initramfs... [ 2.459939] debug: unmapping init [mem 0xffff9d8e3cc54000-0xffff9d8e3ffbffff] [ 2.464132] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.467303] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.470838] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.017889] Initialise system trusted keyrings [ 3.021468] Key type blacklist registered [ 3.024709] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.044408] zbud: loaded [ 3.050347] *** VALIDATE nfs *** [ 3.052482] *** VALIDATE nfs4 *** [ 3.055930] pstore: using deflate compression [ 3.062585] Platform Keyring initialized [ 3.524656] NET: Registered protocol family 38 [ 3.529554] Key type asymmetric registered [ 3.532902] Asymmetric key parser 'x509' registered [ 3.542832] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.549888] io scheduler mq-deadline registered [ 3.555750] io scheduler kyber registered [ 3.562756] io scheduler bfq registered [ 3.569510] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.582979] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.596450] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.622782] ACPI: Power Button [PWRF] [ 3.647890] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.667177] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.701197] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.718560] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.752046] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.802388] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.842709] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.862654] Non-volatile memory driver v1.3 [ 3.866418] Linux agpgart interface v0.103 [ 3.952388] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.964384] vda: detected capacity change from 0 to 74702848 [ 4.012379] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.019153] vdb: detected capacity change from 0 to 1073741824 [ 4.065601] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.074229] vdc: detected capacity change from 0 to 2621440000 [ 4.113359] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.136394] vdd: detected capacity change from 0 to 2621440000 [ 4.181380] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.188937] vde: detected capacity change from 0 to 4294967296 [ 4.239581] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.248474] vdf: detected capacity change from 0 to 4294967296 [ 4.262507] libphy: Fixed MDIO Bus: probed [ 4.283596] usbcore: registered new interface driver usbserial_generic [ 4.292493] usbserial: USB Serial support registered for generic [ 4.299142] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.309538] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.315438] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.320504] mousedev: PS/2 mouse device common for all mice [ 4.332267] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.343501] rtc_cmos 00:05: RTC can wake from S4 [ 4.351945] rtc_cmos 00:05: registered as rtc0 [ 4.363116] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.367418] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.372477] intel_pstate: CPU model not supported [ 4.401952] hid: raw HID events driver (C) Jiri Kosina [ 4.415345] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.419870] usbcore: registered new interface driver usbhid [ 4.419876] usbhid: USB HID core driver [ 4.421350] drop_monitor: Initializing network drop monitor service [ 4.421508] Initializing XFRM netlink socket [ 4.422920] NET: Registered protocol family 10 [ 4.435905] Segment Routing with IPv6 [ 4.472105] NET: Registered protocol family 17 [ 4.474870] mpls_gso: MPLS GSO support [ 4.485545] RAS: Correctable Errors collector initialized. [ 4.488527] AVX version of gcm_enc/dec engaged. [ 4.491193] AES CTR mode by8 optimization enabled [ 4.657158] sched_clock: Marking stable (4657128866, 0)->(5598029309, -940900443) [ 4.666722] registered taskstats version 1 [ 4.671984] Loading compiled-in X.509 certificates [ 4.678085] zswap: loaded using pool lzo/zbud [ 4.742956] Key type big_key registered [ 4.770084] Key type encrypted registered [ 4.773367] ima: No TPM chip found, activating TPM-bypass! [ 4.776897] ima: Allocated hash algorithm: sha1 [ 4.779407] ima: No architecture policies found [ 4.786560] evm: Initialising EVM extended attributes: [ 4.795220] evm: security.selinux [ 4.797437] evm: security.ima [ 4.800538] evm: security.capability [ 4.803415] evm: HMAC attrs: 0x1 [ 4.808373] rtc_cmos 00:05: setting system clock to 2026-08-31 08:18:46 UTC (1788164326) [ 4.829646] debug: unmapping init [mem 0xffffffff9c203000-0xffffffff9c3fffff] [ 4.838888] debug: unmapping init [mem 0xffffffff9af82000-0xffffffff9b258fff] [ 4.863067] Write protecting the kernel read-only data: 28672k [ 4.873760] debug: unmapping init [mem 0xffffffff99603000-0xffffffff997fffff] [ 4.877995] debug: unmapping init [mem 0xffffffff99f14000-0xffffffff99ffffff] [ 4.978433] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.014400] systemd[1]: Detected virtualization kvm. [ 5.020456] systemd[1]: Detected architecture x86-64. [ 5.024797] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.086161] systemd[1]: No hostname configured. [ 5.092445] systemd[1]: Set hostname to . [ 5.096273] random: systemd: uninitialized urandom read (16 bytes read) [ 5.100394] systemd[1]: Initializing machine ID from random generator. [ 5.239132] random: ln: uninitialized urandom read (6 bytes read) [ 5.403743] random: systemd: uninitialized urandom read (16 bytes read) [ 5.407826] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.413777] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.423415] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.918988] device-mapper: uevent: version 1.0.3 [ 6.921633] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.501728] virtio_net virtio0 ens2: renamed from eth0 [ 8.538053] random: fast init done [ 8.718448] scsi host0: ata_piix [ 8.738434] scsi host1: ata_piix [ 8.742832] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.749330] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.332723] random: crng init done [ 14.338334] random: 7 urandom warning(s) missed due to ratelimiting [ 14.442783] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.495190] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.888252] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.742330] SELinux: Disabled at runtime. [ 18.878303] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.894894] systemd[1]: Detected virtualization kvm. [ 18.905530] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.981216] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.988806] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.011560] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.018334] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.023853] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.042444] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.091978] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ 20.423642] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.240769] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.173041] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.179169] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.444773] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.479821] EDAC sbridge: Ver: 1.1.2 [ 25.401257] Key type dns_resolver registered [ 26.013431] NFS: Registering the id_resolver key type [ 26.020317] Key type id_resolver registered [ 26.024580] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ 36.508007] hrtimer: interrupt took 2266505 ns Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg640-server login: [ 73.049751] libcfs: loading out-of-tree module taints kernel. [ 73.096639] Key type ._llcrypt registered [ 73.101384] Key type .llcrypt registered [ 73.187715] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 87.853451] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 88.962758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.975752] alg: No test for adler32 (adler32-zlib) [ 90.259300] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 90.885555] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 92.586347] Key type lgssc registered [ 93.988540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 109.231058] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 139.725394] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 150.476171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 150.530081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 151.834661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 151.875162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 152.000675] Lustre: lustre-MDT0000: new disk, initializing [ 152.071475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.087950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 156.478134] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.024870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.112864] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.139706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.148165] Lustre: Skipped 1 previous similar message [ 169.223384] Lustre: lustre-MDT0001: new disk, initializing [ 169.280948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 169.325155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 169.340467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 173.261928] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.589708] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 185.367994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.594266] Lustre: lustre-OST0000: new disk, initializing [ 185.600297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.606924] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 185.667368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.645957] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.541138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.546915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.629585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 203.550163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.718371] Lustre: lustre-OST0001: new disk, initializing [ 203.725529] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 203.730115] Lustre: 9518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.836948] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.293806] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.504746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 212.522477] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.578309] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 222.470733] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.641140] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 238.814419] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing check_logdir /tmp/testlogs/ [ 243.805648] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing yml_node [ 247.833328] Lustre: DEBUG MARKER: Client: 2.17.56.54 [ 249.488925] Lustre: DEBUG MARKER: MDS: 2.17.56.54 [ 251.227429] Lustre: DEBUG MARKER: OSS: 2.17.56.54 [ 252.694774] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 04:22:53 EDT 2026 [ 265.676634] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 266.914286] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 282.081778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 282.091256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.104781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.130588] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.131186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.137637] Lustre: Skipped 2 previous similar messages [ 284.151133] Lustre: Skipped 3 previous similar messages [ 288.011337] Lustre: server umount lustre-MDT0000 complete [ 291.499405] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164613 with bad export cookie 5010314711727463468 [ 291.507859] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 291.516083] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 294.379717] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.387061] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.393915] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.397808] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 294.423553] Lustre: Skipped 1 previous similar message [ 296.419936] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.425763] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 296.446129] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 296.453794] Lustre: Skipped 2 previous similar messages [ 298.079572] Lustre: server umount lustre-MDT0001 complete [ 302.051304] Lustre: server umount lustre-OST0000 complete [ 305.531717] Lustre: server umount lustre-OST0001 complete [ 311.246719] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 319.610411] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 353.511704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 353.747825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 353.786634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 353.865727] Lustre: lustre-MDT0000: new disk, initializing [ 353.944622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 353.957916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 357.314215] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 366.411337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 366.501653] Lustre: 15069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 366.561905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 366.565486] Lustre: Skipped 1 previous similar message [ 366.629554] Lustre: lustre-MDT0001: new disk, initializing [ 366.729509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 366.791806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 366.807741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 370.008258] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 377.583896] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 382.888817] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 388.670949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 389.050416] Lustre: lustre-OST0000: new disk, initializing [ 389.055188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 389.061677] Lustre: 17040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 389.156475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.917196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 390.930847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 391.009989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 394.396195] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 402.573343] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 406.563399] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 406.764300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 410.981167] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 411.164543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 416.742079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 416.744227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 416.748175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 416.755218] Lustre: Skipped 1 previous similar message [ 426.980084] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 426.984478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 427.002587] Lustre: Skipped 4 previous similar messages [ 427.138743] Lustre: server umount lustre-OST0000 complete [ 432.100768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.742728] Lustre: server umount lustre-MDT0000 complete [ 437.219466] LustreError: 15985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 437.250050] LustreError: 15985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 440.173446] LustreError: 15061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164761 with bad export cookie 5010314711727465001 [ 440.176657] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 440.184841] LustreError: 15061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 440.501609] Lustre: server umount lustre-MDT0001 complete [ 443.362270] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 04:26:03 (1788164763) [ 449.589256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.078334] LustreError: 19305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.166510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 454.418384] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 455.654046] LustreError: 19306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.412030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.214284] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 474.714385] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 481.457888] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 488.833936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.925188] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 501.878154] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 519.149285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 519.165730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.205168] Lustre: Skipped 1 previous similar message [ 519.224505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.236540] Lustre: Skipped 2 previous similar messages [ 523.259410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 524.711752] Lustre: server umount lustre-OST0000 complete [ 534.675337] Lustre: server umount lustre-MDT0000 complete [ 538.598699] LustreError: 20033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.617252] LustreError: 20033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 538.932887] LustreError: 20041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164860 with bad export cookie 5010314711727465883 [ 538.947274] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.950304] LustreError: 20041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 539.540972] Lustre: server umount lustre-MDT0001 complete [ 552.484392] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 555.054921] Key type lgssc unregistered [ 555.467065] LNet: 22860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.473153] LNetError: 22860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 555.501801] LNet: Removed LNI 192.168.206.140@tcp [ 556.460146] Key type .llcrypt unregistered [ 556.464044] Key type ._llcrypt unregistered [ 581.885643] Key type ._llcrypt registered [ 581.888682] Key type .llcrypt registered [ 581.976590] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 583.163321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 583.255851] alg: No test for adler32 (adler32-zlib) [ 584.353548] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 584.579353] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 586.280168] Key type lgssc registered [ 587.684693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 597.875535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 597.888250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.437060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 604.338759] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 609.776593] LustreError: 24273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.798753] LustreError: 24273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 613.097121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.462716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 617.918410] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 625.172406] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 630.559347] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 636.890309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 637.281771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 643.467689] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 644.513506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 650.335875] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 665.058103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 665.064153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.072876] LustreError: Skipped 1 previous similar message [ 665.074030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 665.081476] Lustre: Skipped 1 previous similar message [ 665.101017] Lustre: Skipped 1 previous similar message [ 666.295603] Lustre: server umount lustre-OST0000 complete [ 670.186081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.194699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 670.198374] Lustre: Skipped 1 previous similar message [ 675.300866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 675.313772] Lustre: Skipped 2 previous similar messages [ 676.447139] Lustre: server umount lustre-MDT0000 complete [ 680.274641] LustreError: 24255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165001 with bad export cookie 11314948286489360479 [ 680.283576] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 680.297284] LustreError: 24255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 680.576106] Lustre: server umount lustre-MDT0001 complete [ 691.670242] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 694.910768] Key type lgssc unregistered [ 695.280330] LNet: 28252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.285589] LNetError: 28252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 695.305852] LNet: Removed LNI 192.168.206.140@tcp [ 696.384568] Key type .llcrypt unregistered [ 696.387138] Key type ._llcrypt unregistered [ 720.853599] Key type ._llcrypt registered [ 720.857348] Key type .llcrypt registered [ 721.023321] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 722.306502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 722.323893] alg: No test for adler32 (adler32-zlib) [ 723.294429] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 723.489234] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 725.184275] Key type lgssc registered [ 726.569213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 732.939819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 732.964288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 734.347967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.052120] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 744.934834] LustreError: 29675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 744.950119] LustreError: 29675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 745.429651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.709298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 749.454374] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 755.597990] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 760.312866] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 772.403046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 772.978762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 778.545662] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 780.147307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 785.456571] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 799.207563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 799.219083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 799.239909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 800.744279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 800.745370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 800.749631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.782118] Lustre: Skipped 1 previous similar message [ 803.149374] Lustre: server umount lustre-OST0000 complete [ 807.396831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 807.403731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 807.425320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 810.977990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 810.994943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.881987] Lustre: server umount lustre-MDT0000 complete [ 816.099079] LustreError: 29674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 816.137458] LustreError: 29674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 817.241274] LustreError: 29656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165138 with bad export cookie 1112778508273553841 [ 817.248022] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 817.255901] LustreError: 29656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 817.706815] Lustre: server umount lustre-MDT0001 complete [ 831.155657] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 833.689195] Key type lgssc unregistered [ 834.023507] LNet: 33556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.033950] LNetError: 33556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.053556] LNet: Removed LNI 192.168.206.140@tcp [ 835.099094] Key type .llcrypt unregistered [ 835.102909] Key type ._llcrypt unregistered [ 857.503944] Key type ._llcrypt registered [ 857.506207] Key type .llcrypt registered [ 857.603285] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 858.768177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 858.833171] alg: No test for adler32 (adler32-zlib) [ 859.789808] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 860.005920] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 861.664156] Key type lgssc registered [ 862.691333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 869.271677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 869.285715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.628694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.350322] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 881.125473] LustreError: 34968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.154307] LustreError: 34968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 882.153847] LustreError: 34967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 883.030091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 883.448041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 887.459572] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 895.101631] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 902.200817] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 911.072265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 911.588872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.726236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 918.200092] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 925.604693] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 937.443653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 937.462403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 937.478467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 939.490037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 939.502897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 939.504690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 943.197268] Lustre: server umount lustre-OST0000 complete [ 949.734687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.740856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 949.748496] Lustre: Skipped 1 previous similar message [ 949.760077] Lustre: Skipped 1 previous similar message [ 952.681477] Lustre: server umount lustre-MDT0000 complete [ 954.855906] LustreError: 34967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 955.587183] LustreError: 34950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165277 with bad export cookie 12195422157761865443 [ 955.601559] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 955.823803] Lustre: server umount lustre-MDT0001 complete [ 966.025458] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 970.342350] Key type lgssc unregistered [ 970.763453] LNet: 38506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.767394] LNetError: 38506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.814654] LNet: Removed LNI 192.168.206.140@tcp [ 972.682499] Key type .llcrypt unregistered [ 972.686686] Key type ._llcrypt unregistered [ 997.253218] Key type ._llcrypt registered [ 997.254632] Key type .llcrypt registered [ 997.375144] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 998.498994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.562359] alg: No test for adler32 (adler32-zlib) [ 999.733954] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 1000.027080] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1001.752550] Key type lgssc registered [ 1002.993728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1010.212989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1010.230048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.658340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.232831] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1021.923347] LustreError: 39920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.951732] LustreError: 39920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1022.960766] LustreError: 39921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1025.294246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.631472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1029.835151] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1036.724484] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1041.885897] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1047.075094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1047.374918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.962756] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1057.680892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1058.582472] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1065.958000] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1065.964926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.982904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1068.003466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1068.007447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1068.013970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.035100] Lustre: Skipped 2 previous similar messages [ 1071.287081] Lustre: lustre-OST0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 1076.412895] Lustre: lustre-OST0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 1076.420413] Lustre: Skipped 2 previous similar messages [ 1079.264130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1079.437705] Lustre: server umount lustre-OST0000 complete [ 1098.724136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.743084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1098.750193] Lustre: Skipped 3 previous similar messages [ 1105.060579] Lustre: server umount lustre-MDT0000 complete [ 1108.575465] LustreError: 39900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165430 with bad export cookie 17943874583701104888 [ 1108.582306] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.593392] LustreError: 39900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1114.934337] Lustre: server umount lustre-MDT0001 complete [ 1125.495908] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1128.055974] Key type lgssc unregistered [ 1128.311608] LNet: 43639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.323461] LNetError: 43639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.344754] LNet: Removed LNI 192.168.206.140@tcp [ 1129.116370] Key type .llcrypt unregistered [ 1129.121350] Key type ._llcrypt unregistered [ 1147.847640] Key type ._llcrypt registered [ 1147.857173] Key type .llcrypt registered [ 1147.947879] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1148.603514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1148.680833] alg: No test for adler32 (adler32-zlib) [ 1149.661548] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 1149.897250] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1151.544908] Key type lgssc registered [ 1152.466233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1159.023830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1159.041644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.350246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.489722] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1170.928364] LustreError: 45028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.945416] LustreError: 45028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1171.255158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.586070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1175.450258] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1181.974422] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1187.120089] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1192.988980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1193.277849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.974779] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1205.282316] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1205.562049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1212.385643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1212.395056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.411164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1213.136258] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 1213.961627] Lustre: server umount lustre-MDT0000 complete [ 1215.975697] LustreError: 45756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.975975] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.986490] LustreError: 45756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1216.009430] Lustre: Skipped 1 previous similar message [ 1217.263823] LustreError: 45010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165538 with bad export cookie 12983607679565472841 [ 1217.271844] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1217.275047] LustreError: 45010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1218.218515] LustreError: 45025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.234677] LustreError: 45025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1218.250595] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.40@tcp (stopping) [ 1221.095369] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.101354] LustreError: 45025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.107735] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1223.648978] Lustre: server umount lustre-MDT0001 complete [ 1244.723239] Lustre: server umount lustre-OST0000 complete [ 1261.903395] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1264.597164] Key type lgssc unregistered [ 1264.850530] LNet: 48894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.859813] LNetError: 48894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.879943] LNet: Removed LNI 192.168.206.140@tcp [ 1265.920415] Key type .llcrypt unregistered [ 1265.929169] Key type ._llcrypt unregistered [ 1286.341476] Key type ._llcrypt registered [ 1286.343561] Key type .llcrypt registered [ 1286.440124] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1287.737972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1287.792178] alg: No test for adler32 (adler32-zlib) [ 1288.890432] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 1289.167046] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1290.865065] Key type lgssc registered [ 1291.875599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1298.493243] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1298.503854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1313.952494] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1319.072330] LustreError: 50258:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.140@tcp: failed processing log, type 4: rc = -110 [ 1345.760209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1351.660352] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1360.160556] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1524.662548] Lustre: server umount lustre-OST0000 complete [ 1539.232414] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1541.733565] Key type lgssc unregistered [ 1541.962437] LNet: 51935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.974424] LNetError: 51935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.988897] LNet: Removed LNI 192.168.206.140@tcp [ 1542.674405] Key type .llcrypt unregistered [ 1542.677674] Key type ._llcrypt unregistered [ 1563.599333] Key type ._llcrypt registered [ 1563.602577] Key type .llcrypt registered [ 1563.719247] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1564.355884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1564.365487] alg: No test for adler32 (adler32-zlib) [ 1565.427886] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 1565.797301] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1567.504307] Key type lgssc registered [ 1568.768142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1577.593079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1577.617370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.076203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.357695] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1589.225244] LustreError: 53341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1589.262615] LustreError: 53341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1593.196499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1593.712210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1598.934412] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1608.217294] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1615.559623] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1623.108392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1623.563479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1628.668871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1631.305294] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1638.461294] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1644.008660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1644.009711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.016566] LustreError: Skipped 1 previous similar message [ 1644.018565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1644.037535] Lustre: Skipped 1 previous similar message [ 1646.792243] Lustre: server umount lustre-OST0000 complete [ 1650.145495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1650.151342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1650.178034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.182161] Lustre: Skipped 1 previous similar message [ 1654.242031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1654.242778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1654.252960] Lustre: Skipped 1 previous similar message [ 1656.177966] Lustre: server umount lustre-MDT0000 complete [ 1659.373149] LustreError: 53342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1659.393527] LustreError: 53342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1659.883986] LustreError: 53321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165981 with bad export cookie 5855284069139250395 [ 1659.890449] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1659.902737] LustreError: 53321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1660.397634] Lustre: server umount lustre-MDT0001 complete [ 1670.919377] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1673.044649] Key type lgssc unregistered [ 1673.247828] LNet: 56875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1673.254514] LNetError: 56875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1673.270854] LNet: Removed LNI 192.168.206.140@tcp [ 1673.971837] Key type .llcrypt unregistered [ 1673.976443] Key type ._llcrypt unregistered [ 1696.323308] Key type ._llcrypt registered [ 1696.325837] Key type .llcrypt registered [ 1696.427616] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1697.475761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1697.634859] alg: No test for adler32 (adler32-zlib) [ 1698.708530] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 1699.035200] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1700.792372] Key type lgssc registered [ 1701.843460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1708.445431] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1708.476729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1724.064570] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1729.184650] LustreError: 58261:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.140@tcp: failed processing log, type 4: rc = -110 [ 1755.876086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.814892] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1769.729715] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1776.320433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1777.096650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1777.204837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1782.241920] LustreError: 58286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1782.270185] LustreError: 58286:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1782.358174] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1787.365730] LustreError: 58287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.435264] LustreError: 59298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.453239] LustreError: 59298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1792.531195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.952285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1798.030335] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1805.381156] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1810.879965] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1812.963071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1812.970226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1812.985144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1813.477427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1813.479792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1813.498640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1813.513075] Lustre: Skipped 1 previous similar message [ 1818.601052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1823.720766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1823.731651] Lustre: Skipped 1 previous similar message [ 1827.297882] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1827.435795] Lustre: server umount lustre-OST0000 complete [ 1849.314301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.319312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1849.334720] Lustre: Skipped 1 previous similar message [ 1849.343381] Lustre: Skipped 1 previous similar message [ 1852.735590] Lustre: server umount lustre-MDT0000 complete [ 1853.475751] LustreError: 60289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1853.504452] LustreError: 60289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1856.175815] LustreError: 61001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166177 with bad export cookie 14280453167724381220 [ 1856.192420] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1862.525803] Lustre: server umount lustre-MDT0001 complete [ 1872.223671] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1874.534711] Key type lgssc unregistered [ 1874.839646] LNet: 62007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.847693] LNetError: 62007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.872454] LNet: Removed LNI 192.168.206.140@tcp [ 1875.777519] Key type .llcrypt unregistered [ 1875.781723] Key type ._llcrypt unregistered [ 1895.283615] Key type ._llcrypt registered [ 1895.286543] Key type .llcrypt registered [ 1895.384965] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1896.191470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1896.233172] alg: No test for adler32 (adler32-zlib) [ 1897.240298] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 1897.453649] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1899.128165] Key type lgssc registered [ 1900.557808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1908.131966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1908.157270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.702996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1913.540925] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1919.974697] LustreError: 63403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1920.001928] LustreError: 63403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1923.120234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.617556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1928.451172] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1935.346563] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1940.781123] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1946.376203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1946.739557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.278849] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1954.930830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1960.189477] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1979.880231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1979.888958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.901276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1980.898509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1980.906514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1980.909861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.923739] Lustre: Skipped 1 previous similar message [ 1982.851215] Lustre: server umount lustre-OST0000 complete [ 1990.113148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1990.121721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.132165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1992.784453] Lustre: server umount lustre-MDT0000 complete [ 1995.909987] LustreError: 63383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166317 with bad export cookie 18318729575182993475 [ 1995.909987] LustreError: 63385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166317 with bad export cookie 18318729575182993475 [ 1995.912606] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1995.922307] LustreError: 63385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1996.361336] Lustre: server umount lustre-MDT0001 complete [ 2006.013751] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2008.869510] Key type lgssc unregistered [ 2009.156640] LNet: 66962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.162794] LNetError: 66962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.178622] LNet: Removed LNI 192.168.206.140@tcp [ 2009.950787] Key type .llcrypt unregistered [ 2009.957455] Key type ._llcrypt unregistered [ 2030.158652] Key type ._llcrypt registered [ 2030.160445] Key type .llcrypt registered [ 2030.251665] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2031.056475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2031.176960] alg: No test for adler32 (adler32-zlib) [ 2032.287919] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 2032.483976] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2034.136131] Key type lgssc registered [ 2035.005346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2043.916580] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 04:52:44 (1788166364) [ 2050.188908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2050.208776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.466174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.570849] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2061.799216] LustreError: 68646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.819798] LustreError: 68646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2064.952829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.451749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.080361] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2076.937658] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2082.564975] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2089.118610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2089.458367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.214605] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2096.633452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2101.441182] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2107.502322] Lustre: Failing over lustre-MDT0000 [ 2107.786852] Lustre: server umount lustre-MDT0000 complete [ 2109.124080] LustreError: 68641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2111.482105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2111.501538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2111.516208] LustreError: 69648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.216830] LustreError: 68640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.236063] LustreError: 68640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2116.956594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.101370] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2117.228593] Lustre: *** cfs_fail_loc=135, val=0*** [ 2117.312051] LustreError: 71731:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2117.317223] LustreError: 71731:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.140@tcp: cfg command failed: rc = -2 [ 2117.327245] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2117.327245] [ 2117.340982] LustreError: MGC192.168.206.140@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2117.360296] LustreError: 71720:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2117.373483] LustreError: 71720:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2117.386668] LustreError: 71720:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2117.395169] LustreError: 71720:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2117.416667] Lustre: server umount lustre-MDT0000 complete [ 2117.422700] LustreError: 71720:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2119.346794] LustreError: 70552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.364953] LustreError: 70552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2123.778993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.973251] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2124.258958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2124.318511] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2128.530849] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2129.383990] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2129.390833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2129.621826] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2129.654468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2136.559304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2136.578556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.605108] Lustre: Skipped 2 previous similar messages [ 2136.618463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.620544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.622420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.500758] Lustre: server umount lustre-OST0000 complete [ 2149.859234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.877260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.887057] Lustre: Skipped 1 previous similar message [ 2149.902937] Lustre: Skipped 2 previous similar messages [ 2152.295061] Lustre: server umount lustre-MDT0000 complete [ 2154.981873] LustreError: 68646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.017533] LustreError: 68646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2156.207189] LustreError: 68627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166477 with bad export cookie 14751782418190028635 [ 2156.208642] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2156.229099] LustreError: 68627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2156.623318] Lustre: server umount lustre-MDT0001 complete [ 2166.749293] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2169.576836] Key type lgssc unregistered [ 2169.910764] LNet: 73511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.923879] LNetError: 73511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.943741] LNet: Removed LNI 192.168.206.140@tcp [ 2170.680112] Key type .llcrypt unregistered [ 2170.682499] Key type ._llcrypt unregistered [ 2193.478865] Key type ._llcrypt registered [ 2193.490769] Key type .llcrypt registered [ 2193.607838] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2194.878744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2194.902453] alg: No test for adler32 (adler32-zlib) [ 2195.922253] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 2196.107208] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2197.744213] Key type lgssc registered [ 2198.969394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2207.037817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2207.057438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2208.455869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2212.363721] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2218.978129] LustreError: 74925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2219.001156] LustreError: 74925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2220.076253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.393935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2224.010727] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2230.145228] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2235.596095] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2242.180145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2242.814802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.873534] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2254.077609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2255.628249] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2261.849818] Lustre: Failing over lustre-MDT0000 [ 2262.055992] Lustre: server umount lustre-MDT0000 complete [ 2263.242738] LustreError: 74921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.547465] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.558845] LustreError: 74919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.574274] Lustre: Skipped 1 previous similar message [ 2264.594352] LustreError: 74919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2268.342484] LustreError: 74919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.577499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.679648] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2271.802129] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2271.965661] LustreError: 78015:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2271.977318] LustreError: 78015:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.140@tcp: cfg command failed: rc = -9 [ 2272.000426] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2272.000426] [ 2272.008798] LustreError: MGC192.168.206.140@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2272.023783] LustreError: 78002:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2272.034885] LustreError: 78002:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2272.047887] LustreError: 78002:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2272.055973] LustreError: 78002:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2272.070367] Lustre: server umount lustre-MDT0000 complete [ 2272.073269] LustreError: 78002:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2273.451027] LustreError: 74921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.473135] LustreError: 74921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2277.878796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.007528] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2278.016047] LustreError: 74916:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2278.244108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.300195] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2278.572421] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2282.160537] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2283.525581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2283.573889] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2283.617985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2292.193248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2292.214568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.235429] Lustre: Skipped 1 previous similar message [ 2292.255618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2293.734977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2293.736559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.599052] Lustre: server umount lustre-OST0000 complete [ 2303.975110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.977570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2303.979397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2304.003688] Lustre: Skipped 1 previous similar message [ 2304.013232] Lustre: Skipped 1 previous similar message [ 2305.414756] Lustre: server umount lustre-MDT0000 complete [ 2309.096020] LustreError: 77477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.129161] LustreError: 77477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2309.376547] LustreError: 74904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166631 with bad export cookie 12709707570820406345 [ 2309.393153] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2309.703332] Lustre: server umount lustre-MDT0001 complete [ 2320.644388] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2323.361838] Key type lgssc unregistered [ 2323.616740] LNet: 79799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.626878] LNetError: 79799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.644891] LNet: Removed LNI 192.168.206.140@tcp [ 2324.395089] Key type .llcrypt unregistered [ 2324.396951] Key type ._llcrypt unregistered [ 2336.985859] Key type ._llcrypt registered [ 2336.988140] Key type .llcrypt registered [ 2337.084716] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2337.398440] Key type .llcrypt unregistered [ 2337.401338] Key type ._llcrypt unregistered [ 2356.903792] Key type ._llcrypt registered [ 2356.906083] Key type .llcrypt registered [ 2357.005136] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2357.963666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2357.983219] alg: No test for adler32 (adler32-zlib) [ 2359.030444] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 2359.223091] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2360.873332] Key type lgssc registered [ 2361.860925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2368.392653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2368.408849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.854863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2373.079298] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2380.221991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.258752] LustreError: 81684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.282797] LustreError: 81684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2380.516586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2384.033512] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2389.775241] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2394.339267] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2399.288764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2399.687111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.088848] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2414.158794] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2415.097370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2425.318301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.323470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2425.323980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2425.330923] Lustre: Skipped 1 previous similar message [ 2425.371565] Lustre: Skipped 1 previous similar message [ 2430.438177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.451289] Lustre: Skipped 1 previous similar message [ 2431.008548] Lustre: server umount lustre-OST0000 complete [ 2435.555359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.578088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2435.586110] Lustre: Skipped 1 previous similar message [ 2440.678571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2440.684588] Lustre: Skipped 1 previous similar message [ 2445.793548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2445.808570] Lustre: Skipped 1 previous similar message [ 2449.376110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2449.516793] Lustre: server umount lustre-MDT0000 complete [ 2450.913592] LustreError: 81683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.947073] LustreError: 81683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2452.932217] LustreError: 81663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166774 with bad export cookie 12603487429025215051 [ 2452.938951] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2452.945977] LustreError: 81663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2453.455022] Lustre: server umount lustre-MDT0001 complete [ 2463.284847] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2466.392895] Key type lgssc unregistered [ 2466.713073] LNet: 85218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2466.723511] LNetError: 85218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2466.736977] LNet: Removed LNI 192.168.206.140@tcp [ 2467.594611] Key type .llcrypt unregistered [ 2467.598935] Key type ._llcrypt unregistered [ 2487.398784] Key type ._llcrypt registered [ 2487.401293] Key type .llcrypt registered [ 2487.515657] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2488.494017] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2488.547536] alg: No test for adler32 (adler32-zlib) [ 2489.622073] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 2489.868153] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2491.520155] Key type lgssc registered [ 2492.443071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2499.045289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2499.054641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.504799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2504.019562] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2510.822516] LustreError: 86609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.839170] LustreError: 86609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2511.011856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2511.380370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2515.109426] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2521.869527] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2527.536695] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2534.942857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2535.350695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2541.442403] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2545.659291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2547.772826] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2555.878559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2555.894885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2555.921693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2555.925080] Lustre: Skipped 1 previous similar message [ 2560.877161] Lustre: server umount lustre-OST0000 complete [ 2566.113783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.121738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2566.152651] Lustre: Skipped 2 previous similar messages [ 2571.233814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2571.248517] Lustre: Skipped 2 previous similar messages [ 2575.332618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2575.343696] Lustre: Skipped 1 previous similar message [ 2578.400107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2578.557373] Lustre: server umount lustre-MDT0000 complete [ 2580.451288] LustreError: 86610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.472329] LustreError: 86610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2582.155859] LustreError: 86591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166903 with bad export cookie 17965642233551622313 [ 2582.162516] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.174135] LustreError: 86591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2582.587455] Lustre: server umount lustre-MDT0001 complete [ 2593.174389] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2596.954633] Key type lgssc unregistered [ 2597.202339] LNet: 90159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2597.214186] LNetError: 90159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2597.226661] LNet: Removed LNI 192.168.206.140@tcp [ 2598.342625] Key type .llcrypt unregistered [ 2598.348886] Key type ._llcrypt unregistered [ 2619.878384] Key type ._llcrypt registered [ 2619.880772] Key type .llcrypt registered [ 2619.966579] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2620.743152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2620.816800] alg: No test for adler32 (adler32-zlib) [ 2621.819299] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 2622.022864] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2623.656159] Key type lgssc registered [ 2624.522975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2630.560623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2630.613801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.935443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.354875] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2641.386019] LustreError: 91574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.395228] LustreError: 91574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2641.639807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.920400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2645.679334] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2651.181892] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2655.639703] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2661.626388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2662.049105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.020338] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2675.163570] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2676.344851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2692.071165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.085039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2692.089818] Lustre: Skipped 1 previous similar message [ 2692.824283] Lustre: server umount lustre-OST0000 complete [ 2697.188349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.203273] Lustre: Skipped 1 previous similar message [ 2697.215938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2697.223219] Lustre: Skipped 1 previous similar message [ 2699.110504] Lustre: server umount lustre-MDT0000 complete [ 2702.306401] LustreError: 91571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.323496] LustreError: 91571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2702.631970] LustreError: 91556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167024 with bad export cookie 7461899957309484513 [ 2702.641729] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2702.665028] LustreError: 91556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2705.039734] Lustre: server umount lustre-MDT0001 complete [ 2714.775847] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2717.314212] Key type lgssc unregistered [ 2717.584581] LNet: 95129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2717.592766] LNetError: 95129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2718.637167] LNet: Removed LNI 192.168.206.140@tcp [ 2719.462508] Key type .llcrypt unregistered [ 2719.465191] Key type ._llcrypt unregistered [ 2739.109223] Key type ._llcrypt registered [ 2739.112394] Key type .llcrypt registered [ 2739.257350] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2740.222234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2740.322834] alg: No test for adler32 (adler32-zlib) [ 2741.378310] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 2741.602522] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2743.240596] Key type lgssc registered [ 2744.458409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2751.617565] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2751.660871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2767.264436] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2772.385078] LustreError: 96494:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.140@tcp: failed processing log, type 4: rc = -110 [ 2799.200226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2805.238860] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2812.296147] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2818.521886] Lustre: server umount lustre-OST0000 complete [ 2826.491331] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 05:05:46 (1788167146) [ 2832.411724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.887782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.078102] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2837.999968] LustreError: 98133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2838.019147] LustreError: 98133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2843.105823] LustreError: 98132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2845.319558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2845.671669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2849.897751] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2856.735466] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2861.829452] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2867.887525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.228824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.348486] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2877.496834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2880.848636] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2891.752310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2891.761360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.772708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2892.772914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2892.779088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2892.791092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.307257] Lustre: server umount lustre-OST0000 complete [ 2901.985295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2901.989788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.011697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2902.022527] Lustre: Skipped 1 previous similar message [ 2905.942963] Lustre: server umount lustre-MDT0000 complete [ 2908.134153] LustreError: 98132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.150694] LustreError: 98132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2909.258538] LustreError: 98112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167230 with bad export cookie 14066325418722594765 [ 2909.268109] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2909.269657] LustreError: 98112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2909.621485] Lustre: server umount lustre-MDT0001 complete [ 2919.264404] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2922.037473] Key type lgssc unregistered [ 2922.349503] LNet: 101827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2922.357353] LNetError: 101827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2922.377988] LNet: Removed LNI 192.168.206.140@tcp [ 2923.037348] Key type .llcrypt unregistered [ 2923.039682] Key type ._llcrypt unregistered [ 2958.072505] Key type ._llcrypt registered [ 2958.074836] Key type .llcrypt registered [ 2958.161936] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2959.132308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2959.190531] alg: No test for adler32 (adler32-zlib) [ 2960.313778] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 2960.519050] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2962.216218] Key type lgssc registered [ 2963.341154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2969.250894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2969.274788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.695395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.374541] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2980.837959] LustreError: 104014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.857682] LustreError: 104014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2982.115716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.576419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2986.906139] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2992.806794] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2996.960403] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3002.418264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3002.746905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.196936] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3013.555707] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3016.043475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3026.403079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3026.413437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.417034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3026.434499] Lustre: Skipped 1 previous similar message [ 3030.360561] Lustre: server umount lustre-OST0000 complete [ 3033.570171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3033.583209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.601395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3033.609619] Lustre: Skipped 1 previous similar message [ 3036.642747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.645269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.668927] Lustre: Skipped 1 previous similar message [ 3039.590154] Lustre: server umount lustre-MDT0000 complete [ 3041.762465] LustreError: 104015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3041.780899] LustreError: 104015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3042.767244] LustreError: 106795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167364 with bad export cookie 4657966697433439001 [ 3042.776175] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3042.780437] LustreError: 106795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3043.062848] Lustre: server umount lustre-MDT0001 complete [ 3051.286380] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 3053.559083] Key type lgssc unregistered [ 3053.806574] LNet: 107549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3053.811172] LNetError: 107549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3053.828972] LNet: Removed LNI 192.168.206.140@tcp [ 3054.542057] Key type .llcrypt unregistered [ 3054.546141] Key type ._llcrypt unregistered [ 3068.079465] Key type ._llcrypt registered [ 3068.081337] Key type .llcrypt registered [ 3068.150089] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3068.739064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3068.749839] alg: No test for adler32 (adler32-zlib) [ 3069.748691] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 3069.922836] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 3071.560166] Key type lgssc registered [ 3072.370630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3078.227813] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3078.240979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3093.796355] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3098.912361] LustreError: 108770:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.140@tcp: failed processing log, type 4: rc = -110 [ 3125.600203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3130.710707] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3137.432150] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3145.059783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.417690] LustreError: 108106:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000355acc9c x1875029410187520/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3145.806174] LustreError: MGC192.168.206.140@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3145.823346] LustreError: 109797:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3145.833713] LustreError: 109797:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3145.843629] LustreError: 109797:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3145.975575] Lustre: server umount lustre-MDT0000 complete [ 3145.980314] LustreError: 109797:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3162.712331] Lustre: server umount lustre-OST0000 complete [ 3171.097966] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 3179.078061] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3211.195610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3211.475479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3211.506057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3211.579681] Lustre: lustre-MDT0000: new disk, initializing [ 3211.670079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3211.683637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3216.546971] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3226.523818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3226.621866] Lustre: 112589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3226.665589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3226.670904] Lustre: Skipped 1 previous similar message [ 3226.740296] Lustre: lustre-MDT0001: new disk, initializing [ 3226.824845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3226.864119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3226.878163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3230.670138] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3238.810301] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3244.009157] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3250.022973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3250.336992] Lustre: lustre-OST0000: new disk, initializing [ 3250.344654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3250.357484] Lustre: 114554:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3250.495742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3256.801777] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3258.424235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3258.439199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3258.535046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3265.975973] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3270.399479] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3270.617912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3274.597751] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3274.841388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3279.335447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.335824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3279.349305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3279.357838] LustreError: Skipped 1 previous similar message [ 3279.384012] Lustre: Skipped 1 previous similar message [ 3282.769623] Lustre: server umount lustre-OST0000 complete [ 3288.546088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3288.559310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.578014] Lustre: Skipped 1 previous similar message [ 3288.585314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.570959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.586135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.593745] Lustre: Skipped 1 previous similar message [ 3294.689358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.696732] Lustre: Skipped 1 previous similar message [ 3299.814319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.817219] Lustre: Skipped 1 previous similar message [ 3301.344140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3301.518263] Lustre: server umount lustre-MDT0000 complete [ 3305.025623] LustreError: 113382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167626 with bad export cookie 8999432237391629391 [ 3305.038077] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3305.047773] LustreError: 113382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3305.311144] Lustre: server umount lustre-MDT0001 complete [ 3312.773161] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 05:13:53 (1788167633) [ 3316.364925] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3332.524625] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 3340.996297] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3373.984145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.183400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3374.213240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3374.362569] Lustre: lustre-MDT0000: new disk, initializing [ 3374.418658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.431315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3378.786717] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3388.248511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.355363] Lustre: 119459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3388.426977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3388.433414] Lustre: Skipped 1 previous similar message [ 3388.496191] Lustre: lustre-MDT0001: new disk, initializing [ 3388.617435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3388.715395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3388.728257] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3392.077650] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3401.275740] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3407.341123] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3414.208269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3414.550566] Lustre: lustre-OST0000: new disk, initializing [ 3414.554649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3414.561304] Lustre: 121426:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3414.646089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3420.226278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3420.247971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3420.403744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3421.157764] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3432.398390] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3438.005530] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3438.238412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3442.846879] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3443.048801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3446.245752] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3446.248320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3446.266821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3446.311065] Lustre: Skipped 1 previous similar message [ 3459.041299] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3459.189724] Lustre: server umount lustre-OST0000 complete [ 3465.698486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3465.704498] LustreError: Skipped 1 previous similar message [ 3465.707373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.716939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.722759] Lustre: Skipped 5 previous similar messages [ 3468.890172] Lustre: server umount lustre-MDT0000 complete [ 3471.846079] LustreError: 121135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3471.855468] LustreError: 121135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3472.561691] LustreError: 119451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167794 with bad export cookie 8999432237391630273 [ 3472.567974] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3472.587730] LustreError: 119451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3472.881456] Lustre: server umount lustre-MDT0001 complete [ 3482.807295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.837194] LustreError: 123644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3483.894608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.253456] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3489.249223] LustreError: 123645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3493.355369] LustreError: 123644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.472812] LustreError: 123645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.763730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.513639] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3513.547593] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3519.572661] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3525.988424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3532.396425] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3539.826430] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3549.212358] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3554.275359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3554.289121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3554.293126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3554.293146] Lustre: Skipped 2 previous similar messages [ 3554.305033] Lustre: Skipped 2 previous similar messages [ 3559.611792] Lustre: server umount lustre-OST0000 complete [ 3569.590480] Lustre: server umount lustre-MDT0000 complete [ 3569.634920] LustreError: 123641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.656900] LustreError: 123641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3572.555438] LustreError: 123626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167894 with bad export cookie 8999432237391631148 [ 3572.567736] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3572.573963] LustreError: 123626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3572.799661] Lustre: server umount lustre-MDT0001 complete [ 3583.311504] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 3585.746230] Key type lgssc unregistered [ 3586.023941] LNet: 127275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3586.035944] LNetError: 127275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3586.077431] LNet: Removed LNI 192.168.206.140@tcp [ 3586.815805] Key type .llcrypt unregistered [ 3586.817557] Key type ._llcrypt unregistered [ 3602.877829] Key type ._llcrypt registered [ 3602.881184] Key type .llcrypt registered [ 3602.994761] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 3618.601318] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3619.685251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3619.787188] alg: No test for adler32 (adler32-zlib) [ 3621.040686] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 3621.491385] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 3623.272412] Key type lgssc registered [ 3624.820428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3658.074752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3658.117043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3659.322876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3659.349966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3659.426527] Lustre: lustre-MDT0000: new disk, initializing [ 3659.494825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.514840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3663.378979] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3672.616940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3672.715431] Lustre: 130429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3672.770230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3672.774898] Lustre: Skipped 1 previous similar message [ 3672.835795] Lustre: lustre-MDT0001: new disk, initializing [ 3672.915124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3672.942673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3672.952587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3676.695804] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3685.542158] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3691.046300] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3697.478951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3697.813897] Lustre: lustre-OST0000: new disk, initializing [ 3697.816779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3697.822946] Lustre: 132394:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3697.921949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3703.621343] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3705.925266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3705.940347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3706.043773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3712.233699] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3716.897114] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3717.090682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3722.000550] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3722.205525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3726.818102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.845100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3726.861803] Lustre: Skipped 1 previous similar message [ 3726.870889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3726.903912] Lustre: Skipped 1 previous similar message [ 3730.723840] Lustre: server umount lustre-OST0000 complete [ 3737.057452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.111017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3737.128043] Lustre: Skipped 1 previous similar message [ 3742.896668] Lustre: server umount lustre-MDT0000 complete [ 3747.210721] LustreError: 130423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168068 with bad export cookie 11557551761592864244 [ 3747.214426] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3747.226824] LustreError: 130423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3747.772706] Lustre: server umount lustre-MDT0001 complete [ 3754.959462] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:21:15 (1788168075) [ 3760.202626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.598959] LustreError: 134808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3760.617675] LustreError: 134808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3760.667971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3764.220657] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3765.732877] LustreError: 134809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3770.850655] LustreError: 134808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.586232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3772.906569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3777.105761] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3784.185461] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3789.390069] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3791.330135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.344257] Lustre: Skipped 1 previous similar message [ 3791.354566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3791.364498] Lustre: Skipped 1 previous similar message [ 3796.450201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3796.462819] Lustre: Skipped 1 previous similar message [ 3802.594896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3802.600031] Lustre: Skipped 2 previous similar messages [ 3805.153592] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3805.373900] Lustre: server umount lustre-MDT0000 complete [ 3807.717333] LustreError: 134805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3807.754825] LustreError: 134805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3809.065142] LustreError: 136375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168130 with bad export cookie 11557551761592865126 [ 3809.077558] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3809.080382] LustreError: 136375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3809.435877] Lustre: server umount lustre-MDT0001 complete [ 3817.921630] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:22:18 (1788168138) [ 3825.671958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3841.376785] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3846.496704] LustreError: 137221:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.140@tcp: failed processing log, type 4: rc = -110 [ 3872.224411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3877.988784] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3885.346741] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3887.711553] Lustre: server umount lustre-OST0000 complete [ 3896.462816] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:23:36 (1788168216) [ 3902.578347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3902.887640] LustreError: 138667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3902.939487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3907.299567] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3913.199271] LustreError: 138667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3913.218900] LustreError: 138667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3914.450458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.851625] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3925.765118] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3930.455189] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3936.169026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3936.519562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3936.524370] Lustre: Skipped 1 previous similar message [ 3941.306127] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3947.195454] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3958.258711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3958.273354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3958.276115] Lustre: Skipped 3 previous similar messages [ 3958.292150] Lustre: Skipped 2 previous similar messages [ 3963.062311] Lustre: server umount lustre-MDT0000 complete [ 3963.365552] LustreError: 138667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.382133] LustreError: 138667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3965.972335] LustreError: 138649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168287 with bad export cookie 11557551761592865791 [ 3965.984627] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3965.989996] LustreError: 138649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3966.255841] Lustre: server umount lustre-MDT0001 complete [ 3978.620498] Lustre: server umount lustre-OST0000 complete [ 3985.965111] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:25:06 (1788168306) [ 3993.397517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.963908] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3999.202963] LustreError: 142328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3999.221505] LustreError: 142328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4005.335028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4005.680215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4005.693536] Lustre: Skipped 1 previous similar message [ 4009.663001] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4015.988090] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4021.169441] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4026.925738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4032.535895] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4039.033808] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4039.614115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4042.557899] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4042.740795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4046.749502] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4046.939200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4049.894260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.896370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4049.897498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4049.908043] Lustre: Skipped 1 previous similar message [ 4054.839001] Lustre: server umount lustre-OST0000 complete [ 4060.132713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.204291] Lustre: server umount lustre-MDT0000 complete [ 4065.249370] LustreError: 142324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4065.262101] LustreError: 142324:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4066.976994] LustreError: 142311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168388 with bad export cookie 11557551761592866862 [ 4066.981461] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4066.991947] LustreError: 142311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4067.224661] Lustre: server umount lustre-MDT0001 complete [ 4074.068912] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:26:34 (1788168394) [ 4080.279032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4095.841992] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4100.960449] LustreError: 146483:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.140@tcp: failed processing log, type 4: rc = -110 [ 4131.436950] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4137.625864] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4143.129145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.547939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4143.551381] Lustre: Skipped 2 previous similar messages [ 4143.632067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4147.073960] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4153.979149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4157.850696] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4164.085049] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4168.717654] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4173.753523] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4174.080067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4178.561166] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4178.891257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4181.992901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.998634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4182.015361] Lustre: Skipped 2 previous similar messages [ 4182.025410] Lustre: Skipped 3 previous similar messages [ 4195.296112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4195.574093] Lustre: server umount lustre-MDT0000 complete [ 4197.347463] LustreError: 148585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4197.374896] LustreError: 148585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4198.745182] LustreError: 146489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168520 with bad export cookie 11557551761592867884 [ 4198.754275] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4198.765436] LustreError: 146489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4199.093373] Lustre: server umount lustre-MDT0001 complete [ 4212.885537] Lustre: server umount lustre-OST0000 complete [ 4221.361441] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:29:01 (1788168541) [ 4228.391487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4244.065062] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4249.186643] LustreError: 150679:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.140@tcp: failed processing log, type 4: rc = -110 [ 4281.242138] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4288.370421] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4294.693764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4295.258928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4299.000930] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4306.093794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4310.208517] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4316.426465] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4320.989476] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4326.946201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.149255] Lustre: lustre-OST0001: new disk, initializing [ 4327.156416] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4327.169471] Lustre: 153552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4333.577457] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4336.176839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4336.187658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4336.284482] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4342.439040] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4346.349898] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4346.527928] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4350.383233] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4350.536358] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4351.988395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4352.002672] Lustre: Skipped 1 previous similar message [ 4352.010602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4352.013384] Lustre: Skipped 8 previous similar messages [ 4356.579692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4366.306066] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4366.389954] Lustre: server umount lustre-OST0000 complete [ 4372.450163] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4372.460139] LustreError: Skipped 1 previous similar message [ 4376.538931] Lustre: server umount lustre-OST0001 complete [ 4386.316500] Lustre: server umount lustre-MDT0000 complete [ 4390.423398] LustreError: 153551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168712 with bad export cookie 11557551761592868836 [ 4390.427979] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4390.437821] LustreError: 153551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4390.771498] Lustre: server umount lustre-MDT0001 complete [ 4412.387441] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:32:12 (1788168732) [ 4413.820648] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4415.064793] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4416.771732] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:32:17 (1788168737) [ 4422.860983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4423.316852] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4423.354869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4423.633519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.638795] Lustre: Skipped 5 previous similar messages [ 4428.340926] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4436.292816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.398801] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4436.435537] Lustre: 157114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4441.962853] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4450.054072] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4455.868413] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4462.075180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4462.204488] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4467.005912] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4469.889241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4472.691913] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4476.926235] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4477.074482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4480.498964] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4480.667794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4485.089539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4485.096426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.107115] Lustre: Skipped 5 previous similar messages [ 4485.113572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4485.118874] Lustre: Skipped 9 previous similar messages [ 4496.354329] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4496.495242] Lustre: server umount lustre-OST0000 complete [ 4515.986187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4521.493677] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4525.484469] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4525.503470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4525.504159] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4525.525026] Lustre: Skipped 1 previous similar message [ 4528.592818] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4533.104693] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4533.279745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4537.771735] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4538.007117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4543.050210] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff952906cf3800.ost_server_uuid 50 [ 4544.469689] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff952906cf3800.ost_server_uuid in FULL state after 0 sec [ 4544.750838] LustreError: 158119:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.40@tcp) returned error from blocking AST (req@000000006d653a24 x1875029989248384 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000005b60478/0xa064ac17b6eee93e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.40@tcp remote: 0x6ab9159d8ca0d1c0 expref: 5 pid: 157121 timeout: 4644 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4544.786326] LustreError: lustre-MDT0000: A client on nid 192.168.206.40@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4544.798550] LustreError: 157110:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.40@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000005b60478/0xa064ac17b6eee93e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.40@tcp remote: 0x6ab9159d8ca0d1c0 expref: 6 pid: 157121 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4551.137119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4551.150343] LustreError: Skipped 2 previous similar messages [ 4555.607057] Lustre: server umount lustre-OST0000 complete [ 4566.755147] Lustre: server umount lustre-MDT0000 complete [ 4570.596065] LustreError: 157120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4570.628342] LustreError: 157120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 4570.959695] LustreError: 157106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168892 with bad export cookie 11557551761592870019 [ 4570.974649] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4577.364985] Lustre: server umount lustre-MDT0001 complete [ 4589.051774] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 4592.267257] Key type lgssc unregistered [ 4592.638742] LNet: 163231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4592.658811] LNetError: 163231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4592.675526] LNet: Removed LNI 192.168.206.140@tcp [ 4593.649419] Key type .llcrypt unregistered [ 4593.652832] Key type ._llcrypt unregistered [ 4615.405816] Key type ._llcrypt registered [ 4615.410554] Key type .llcrypt registered [ 4615.520097] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4616.306931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4616.338703] alg: No test for adler32 (adler32-zlib) [ 4617.300824] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 4617.530901] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 4619.200148] Key type lgssc registered [ 4620.081676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4626.132111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4626.148479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.494602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4630.873546] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4637.679383] LustreError: 164644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4637.700442] LustreError: 164644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4638.211869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4638.578027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4642.699906] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4649.470552] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4655.304210] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4662.227163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4662.724325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4668.459899] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4673.017950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4677.209327] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4684.415232] Lustre: Failing over lustre-MDT0000 [ 4684.771113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4684.783730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.797846] LustreError: 164644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4684.871027] Lustre: server umount lustre-MDT0000 complete [ 4688.354473] LustreError: 165332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4688.360138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.374542] LustreError: 165332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4688.389179] Lustre: Skipped 1 previous similar message [ 4693.473408] LustreError: 164645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4693.485920] LustreError: 164645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4698.372963] LustreError: 164626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.40@tcp arrived at 1788169020 with bad export cookie 5305066659696061641 [ 4698.612023] LustreError: 164644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4698.635204] LustreError: 164644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4704.736475] Lustre: 163957:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788169010/real 1788169010] req@000000004971dfbf x1875031033407488/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788169026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4704.775963] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4709.862315] LustreError: 164641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4709.898181] LustreError: 164641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4715.712511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4729.316693] LustreError: 163954:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c6d9339f x1875031033415040/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4729.350451] LustreError: 164644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4729.383644] LustreError: 164644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4729.619486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4729.664506] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4733.563346] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4734.949777] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4734.956716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4735.553289] Lustre: lustre-MDT0000: Denying connection for new client f5d16d12-0acc-4e09-b025-d9b928561929 (at 192.168.206.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4745.187494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4745.194259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4745.199917] Lustre: Skipped 1 previous similar message [ 4745.210865] Lustre: Skipped 1 previous similar message [ 4749.247776] Lustre: server umount lustre-OST0000 complete [ 4752.035807] LustreError: 168375:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4752.045254] Lustre: 167724:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4752.052637] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4752.076850] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4752.093107] LustreError: 167724:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001b0ffe8a x1875031033429504/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4752.113385] LustreError: 167724:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4752.123224] LustreError: 167724:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4752.138462] Lustre: lustre-MDT0000: Recovery over after 0:18, of 2 clients 0 recovered and 2 were evicted. [ 4752.366394] Lustre: server umount lustre-MDT0000 complete [ 4755.272640] LustreError: 164625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169076 with bad export cookie 5305066659696061739 [ 4755.286723] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4755.287755] LustreError: 164625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4755.291007] LustreError: 163954:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4761.616392] Lustre: server umount lustre-MDT0001 complete [ 4770.272177] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 4772.710429] Key type lgssc unregistered [ 4772.957934] LNet: 169130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4772.965523] LNetError: 169130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4772.984230] LNet: Removed LNI 192.168.206.140@tcp [ 4773.683978] Key type .llcrypt unregistered [ 4773.686251] Key type ._llcrypt unregistered [ 4793.546551] Key type ._llcrypt registered [ 4793.549212] Key type .llcrypt registered [ 4793.612848] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4794.434561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4794.490186] alg: No test for adler32 (adler32-zlib) [ 4795.480833] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 4795.683744] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 4797.408119] Key type lgssc registered [ 4798.483706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4805.808497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4805.830823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4807.238487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4810.919141] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4816.357042] LustreError: 170521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4816.376023] LustreError: 170521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4817.823190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4818.089313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4821.709192] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4827.411466] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4832.594536] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4838.624299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4838.853479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4845.116404] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4851.808317] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4852.232241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4857.319663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4857.321709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4857.341718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4857.355301] Lustre: Skipped 1 previous similar message [ 4859.128066] Lustre: server umount lustre-OST0000 complete [ 4862.434501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4862.437199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4862.444268] Lustre: Skipped 2 previous similar messages [ 4867.557403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4867.566721] Lustre: Skipped 2 previous similar messages [ 4868.586648] Lustre: server umount lustre-MDT0000 complete [ 4871.798507] LustreError: 170503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169193 with bad export cookie 14126404375301393410 [ 4871.806044] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4871.813983] LustreError: 170503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4872.098991] Lustre: server umount lustre-MDT0001 complete [ 4880.804886] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 4883.040075] Key type lgssc unregistered [ 4883.337438] LNet: 174054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4883.348356] LNetError: 174054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4884.392510] LNet: Removed LNI 192.168.206.140@tcp [ 4885.135047] Key type .llcrypt unregistered [ 4885.142834] Key type ._llcrypt unregistered [ 4911.858631] Key type ._llcrypt registered [ 4911.861540] Key type .llcrypt registered [ 4911.963760] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4912.806797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4912.828874] alg: No test for adler32 (adler32-zlib) [ 4913.796764] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 4913.998599] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 4915.648697] Key type lgssc registered [ 4916.857935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4923.614708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4923.635970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4924.976194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4928.860472] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4935.143738] LustreError: 175735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4935.162426] LustreError: 175735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4936.085400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4936.464309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4939.984929] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4945.836120] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4950.270142] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4955.502831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4955.830746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4961.084621] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4966.359732] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4970.153141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4977.636146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4977.644031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4977.656888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4980.713085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4980.718777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4980.727505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4980.744528] Lustre: Skipped 1 previous similar message [ 4982.188186] Lustre: server umount lustre-OST0000 complete [ 4985.825538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4985.849077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4990.947045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4990.951569] Lustre: Skipped 2 previous similar messages [ 4991.692911] Lustre: server umount lustre-MDT0000 complete [ 4994.782602] LustreError: 175715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169316 with bad export cookie 17818617048212792979 [ 4994.785118] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4994.789653] LustreError: 175715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4995.101409] Lustre: server umount lustre-MDT0001 complete [ 5003.090635] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5005.191740] Key type lgssc unregistered [ 5005.404499] LNet: 179271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5005.410543] LNetError: 179271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5005.429536] LNet: Removed LNI 192.168.206.140@tcp [ 5006.073430] Key type .llcrypt unregistered [ 5006.075647] Key type ._llcrypt unregistered [ 5026.296267] Key type ._llcrypt registered [ 5026.298730] Key type .llcrypt registered [ 5026.390158] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5027.458808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5027.479664] alg: No test for adler32 (adler32-zlib) [ 5028.549936] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 5028.872616] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5030.536161] Key type lgssc registered [ 5031.699777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5041.351950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5041.376218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5042.780392] Lustre: *** cfs_fail_loc=135, val=0*** [ 5042.890904] LustreError: 180795:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5042.900871] LustreError: 180795:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.140@tcp: cfg command failed: rc = -2 [ 5042.914368] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5042.914368] [ 5042.926857] LustreError: MGC192.168.206.140@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5042.960290] LustreError: 180756:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5042.992808] LustreError: 180756:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5043.003312] LustreError: 180756:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5043.009555] LustreError: 180756:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5043.100366] Lustre: server umount lustre-MDT0000 complete [ 5043.106991] LustreError: 180756:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5049.554194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5049.962674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5053.482344] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5059.555593] LustreError: 181166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5059.570870] LustreError: 181166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5060.772616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.080731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5064.434315] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5069.865721] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5074.597766] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5079.522833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5079.539645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5079.545628] Lustre: Skipped 1 previous similar message [ 5083.040366] Lustre: server umount lustre-MDT0000 complete [ 5084.647643] LustreError: 181163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5084.663718] LustreError: 181163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5086.618927] LustreError: 182733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169408 with bad export cookie 13631435704912867596 [ 5086.623241] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5086.635384] LustreError: 182733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5087.198690] Lustre: server umount lustre-MDT0001 complete [ 5097.903978] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5100.741481] Key type lgssc unregistered [ 5101.085165] LNet: 183488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5101.096872] LNetError: 183488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5101.127064] LNet: Removed LNI 192.168.206.140@tcp [ 5102.164632] Key type .llcrypt unregistered [ 5102.166865] Key type ._llcrypt unregistered [ 5121.784882] Key type ._llcrypt registered [ 5121.787391] Key type .llcrypt registered [ 5121.869882] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5122.144412] Key type .llcrypt unregistered [ 5122.145827] Key type ._llcrypt unregistered [ 5138.293703] Key type ._llcrypt registered [ 5138.295523] Key type .llcrypt registered [ 5138.441867] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5139.682693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5139.723938] alg: No test for adler32 (adler32-zlib) [ 5140.734374] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 5140.960760] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5142.632156] Key type lgssc registered [ 5143.545904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5149.424762] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5149.437968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5164.896457] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5170.018809] LustreError: 185624:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.140@tcp: failed processing log, type 4: rc = -110 [ 5196.704462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5202.127715] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5208.266896] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5214.116692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.568556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5214.652125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5218.550487] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5219.812668] LustreError: 185649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5219.832984] LustreError: 185649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5224.930568] LustreError: 185650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5225.648019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.937781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5229.011869] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5234.690709] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5239.101096] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5241.373561] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5255.650496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5255.657448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.672682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5256.678723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5256.685790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5256.698967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5257.699213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5257.705578] Lustre: Skipped 1 previous similar message [ 5259.937909] Lustre: server umount lustre-OST0000 complete [ 5266.915520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5266.924329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.941425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5269.443144] Lustre: server umount lustre-MDT0000 complete [ 5272.657074] LustreError: 185631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169594 with bad export cookie 13384026467612284213 [ 5272.658258] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5272.666266] LustreError: 185631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5272.940706] Lustre: server umount lustre-MDT0001 complete [ 5281.891630] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5284.287700] Key type lgssc unregistered [ 5284.662775] LNet: 189690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5284.674031] LNetError: 189690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5284.695902] LNet: Removed LNI 192.168.206.140@tcp [ 5285.444290] Key type .llcrypt unregistered [ 5285.446252] Key type ._llcrypt unregistered [ 5305.119562] Key type ._llcrypt registered [ 5305.126739] Key type .llcrypt registered [ 5305.227536] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5306.300629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5306.313646] alg: No test for adler32 (adler32-zlib) [ 5307.332368] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 5307.557443] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5309.208250] Key type lgssc registered [ 5310.243557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5316.927256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5316.952081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5318.336909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.503906] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5328.869195] LustreError: 191086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.888086] LustreError: 191086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5330.534616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5330.890281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5334.170921] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5340.021292] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5344.337789] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5349.761371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5350.071142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5355.345616] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5361.801696] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5363.387812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5371.107621] Lustre: Failing over lustre-MDT0000 [ 5371.347674] Lustre: server umount lustre-MDT0000 complete [ 5371.875764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5371.887528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5371.908448] LustreError: 191087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.640432] LustreError: 191082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.923589] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5373.940936] Lustre: Skipped 1 previous similar message [ 5378.729593] LustreError: 191079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5378.755806] LustreError: 191079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5383.847708] LustreError: 191082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5383.867881] LustreError: 191082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5389.433554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5389.539505] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5389.749525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5389.820422] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5390.038327] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5393.903476] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5394.934154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5394.962878] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5395.024769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5398.017348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5401.400546] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5410.289257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5410.303130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5410.308591] Lustre: Skipped 1 previous similar message [ 5414.509550] Lustre: server umount lustre-OST0000 complete [ 5420.516590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5420.517353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5420.537720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5420.582253] Lustre: Skipped 2 previous similar messages [ 5425.635325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5425.646914] Lustre: Skipped 2 previous similar messages [ 5430.754553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5430.759914] Lustre: Skipped 1 previous similar message [ 5432.288191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5432.515589] Lustre: server umount lustre-MDT0000 complete [ 5435.874250] LustreError: 191087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5435.892849] LustreError: 191087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5436.344268] LustreError: 191065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169758 with bad export cookie 14392282622883200472 [ 5436.360899] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5436.367825] LustreError: 191065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5436.878970] Lustre: server umount lustre-MDT0001 complete [ 5447.731210] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5449.966137] Key type lgssc unregistered [ 5450.199969] LNet: 195872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5450.208981] LNetError: 195872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5450.224415] LNet: Removed LNI 192.168.206.140@tcp [ 5450.855255] Key type .llcrypt unregistered [ 5450.858160] Key type ._llcrypt unregistered [ 5472.816683] Key type ._llcrypt registered [ 5472.819626] Key type .llcrypt registered [ 5472.961498] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5474.154493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5474.180160] alg: No test for adler32 (adler32-zlib) [ 5475.387330] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 5475.711898] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5477.352606] Key type lgssc registered [ 5478.493961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5486.460295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5486.482156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5487.853976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5491.496325] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5497.315258] LustreError: 197278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5497.338037] LustreError: 197278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5499.017140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5499.601531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5503.525463] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5509.181384] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5514.713173] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5520.493430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5520.847207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5525.748520] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5532.227599] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5533.177756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5535.963183] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5543.471944] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5559.460770] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5565.925107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5565.932324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5565.947722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5568.996145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5569.003490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5569.007317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.028691] Lustre: Skipped 1 previous similar message [ 5571.393327] Lustre: server umount lustre-OST0000 complete [ 5576.162797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5576.169415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5576.177517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5579.236770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5579.242193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5579.257489] Lustre: Skipped 1 previous similar message [ 5580.701917] Lustre: server umount lustre-MDT0000 complete [ 5583.858834] LustreError: 197260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169905 with bad export cookie 12618555379422041254 [ 5583.863914] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5583.872588] LustreError: 197260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5584.248544] Lustre: server umount lustre-MDT0001 complete [ 5595.163246] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5597.398536] Key type lgssc unregistered [ 5597.704409] LNet: 201145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5597.712884] LNetError: 201145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5598.760346] LNet: Removed LNI 192.168.206.140@tcp [ 5599.675883] Key type .llcrypt unregistered [ 5599.685738] Key type ._llcrypt unregistered [ 5617.597586] Key type ._llcrypt registered [ 5617.599571] Key type .llcrypt registered [ 5617.679135] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5618.414111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5618.467298] alg: No test for adler32 (adler32-zlib) [ 5619.715406] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 5619.991090] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5621.673603] Key type lgssc registered [ 5622.776437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5630.386572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5630.401206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.935073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.043573] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5642.215776] LustreError: 202409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5642.242727] LustreError: 202409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5644.573158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5644.969299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5648.624450] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5654.789140] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5659.642974] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5666.162326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5666.503717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5671.964760] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5676.722062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5678.345661] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5683.251590] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5696.361595] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5702.558338] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5711.322731] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5721.058274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5721.068647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.087756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5723.115255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5723.122489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5723.129086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.830356] Lustre: server umount lustre-OST0000 complete [ 5728.226243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.240777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5728.246469] Lustre: Skipped 2 previous similar messages [ 5733.347329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5733.355352] Lustre: Skipped 1 previous similar message [ 5734.448359] Lustre: server umount lustre-MDT0000 complete [ 5737.620309] LustreError: 202389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170059 with bad export cookie 17769397444092234287 [ 5737.623479] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5737.630563] LustreError: 202389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5737.894803] Lustre: server umount lustre-MDT0001 complete [ 5747.188383] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5750.116291] Key type lgssc unregistered [ 5750.444886] LNet: 207385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5750.451747] LNetError: 207385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5750.472780] LNet: Removed LNI 192.168.206.140@tcp [ 5751.277250] Key type .llcrypt unregistered [ 5751.280192] Key type ._llcrypt unregistered [ 5774.701151] Key type ._llcrypt registered [ 5774.703116] Key type .llcrypt registered [ 5774.816537] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5775.855228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5775.949471] alg: No test for adler32 (adler32-zlib) [ 5776.946752] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 5777.229413] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5778.880188] Key type lgssc registered [ 5779.937466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5787.313479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5787.326726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.794580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.963186] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5798.379612] LustreError: 208796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.397942] LustreError: 208796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5800.836379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.206509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5806.156437] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5813.086944] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5818.067457] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5823.902488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5824.300667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.277522] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5833.595892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5836.511815] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5849.059893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5849.061379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5849.068559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5849.074501] Lustre: Skipped 1 previous similar message [ 5849.086665] Lustre: Skipped 1 previous similar message [ 5854.183151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5854.187853] Lustre: Skipped 1 previous similar message [ 5854.774933] Lustre: server umount lustre-OST0000 complete [ 5859.298444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.303157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5859.307914] Lustre: Skipped 1 previous similar message [ 5859.323483] Lustre: Skipped 1 previous similar message [ 5864.420517] LustreError: 210694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.434724] LustreError: 210694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5864.458298] Lustre: server umount lustre-MDT0000 complete [ 5868.000127] LustreError: 208778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170189 with bad export cookie 12893541901893774786 [ 5868.002608] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5868.014299] LustreError: 208778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5868.238528] Lustre: server umount lustre-MDT0001 complete [ 5877.982280] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5880.463785] Key type lgssc unregistered [ 5880.761179] LNet: 212482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5880.779914] LNetError: 212482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5880.792512] LNet: Removed LNI 192.168.206.140@tcp [ 5881.520985] Key type .llcrypt unregistered [ 5881.527157] Key type ._llcrypt unregistered [ 5897.970615] Key type ._llcrypt registered [ 5897.973183] Key type .llcrypt registered [ 5898.073370] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5899.446960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5899.471204] alg: No test for adler32 (adler32-zlib) [ 5900.542355] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 5900.727846] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5902.424166] Key type lgssc registered [ 5903.428373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5910.376627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5910.392757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.871892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5915.467653] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5921.598512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5922.097453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5926.579446] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5933.871482] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5939.703698] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5946.624212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5946.987545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.960904] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5958.271397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5962.212622] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5977.058525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5977.065638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.076597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5977.096179] Lustre: Skipped 1 previous similar message [ 5977.128366] Lustre: Skipped 1 previous similar message [ 5980.259557] Lustre: server umount lustre-OST0000 complete [ 5983.717657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5983.725502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.744986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5987.297802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5987.298841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5987.341544] Lustre: Skipped 1 previous similar message [ 5989.125069] Lustre: server umount lustre-MDT0000 complete [ 5992.413383] LustreError: 213722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170314 with bad export cookie 9169028715553696336 [ 5992.418814] LustreError: 213740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5992.420482] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5992.424522] LustreError: 213722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5992.463713] LustreError: 213740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5992.741976] Lustre: server umount lustre-MDT0001 complete [ 6002.527376] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6005.309978] Key type lgssc unregistered [ 6005.597497] LNet: 217423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6005.616611] LNetError: 217423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6005.652723] LNet: Removed LNI 192.168.206.140@tcp [ 6006.639288] Key type .llcrypt unregistered [ 6006.642198] Key type ._llcrypt unregistered [ 6026.969284] Key type ._llcrypt registered [ 6026.977113] Key type .llcrypt registered [ 6027.126847] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6028.047853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6028.154581] alg: No test for adler32 (adler32-zlib) [ 6029.262906] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 6029.558339] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 6031.400166] Key type lgssc registered [ 6032.883995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6041.123611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6041.151153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6042.669077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6046.915332] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6052.834814] LustreError: 218691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.849292] LustreError: 218691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6054.505441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6055.035935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6058.835508] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6065.189866] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6069.824756] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6076.093115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6076.569809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6082.243866] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6087.857679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6088.875655] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6096.658694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6096.715548] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6096.907975] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6102.535094] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6109.952736] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6111.311792] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6122.840735] Lustre: Permanently deactivating lustre-OST0001 [ 6122.854302] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6131.257373] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6140.382652] Lustre: Permanently reactivating lustre-OST0001 [ 6140.396589] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6140.403266] Lustre: Skipped 2 previous similar messages [ 6147.631185] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.642704] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6147.648491] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6147.668019] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6147.723368] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6153.192057] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6153.210212] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6153.217337] Lustre: Skipped 2 previous similar messages [ 6153.221879] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6153.227621] Lustre: Skipped 1 previous similar message [ 6157.055613] Lustre: server umount lustre-OST0001 complete [ 6158.321810] LustreError: 221134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.355195] LustreError: 221134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6161.901806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6161.910046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6161.929494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6163.430908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6163.431022] LustreError: 221134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.455916] LustreError: 221134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6168.547520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6168.558243] LustreError: 221134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.562296] Lustre: Skipped 2 previous similar messages [ 6168.589054] LustreError: 221134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6173.682481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.682705] LustreError: 220510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6173.697183] Lustre: Skipped 1 previous similar message [ 6175.200530] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6175.332470] Lustre: server umount lustre-OST0000 complete [ 6178.785902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.795127] Lustre: Skipped 2 previous similar messages [ 6183.909062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6183.915596] Lustre: Skipped 3 previous similar messages [ 6184.648966] Lustre: server umount lustre-MDT0000 complete [ 6188.302210] LustreError: 218672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170509 with bad export cookie 696139936390366708 [ 6188.307500] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6188.316871] LustreError: 218672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6188.564650] Lustre: server umount lustre-MDT0001 complete [ 6201.179617] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6203.958070] Key type lgssc unregistered [ 6204.300813] LNet: 223797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6204.318895] LNetError: 223797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6204.349853] LNet: Removed LNI 192.168.206.140@tcp [ 6205.061941] Key type .llcrypt unregistered [ 6205.065240] Key type ._llcrypt unregistered [ 6240.993586] Key type ._llcrypt registered [ 6240.998490] Key type .llcrypt registered [ 6241.088994] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6242.121992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6242.164212] alg: No test for adler32 (adler32-zlib) [ 6243.173480] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 6243.393589] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 6245.048155] Key type lgssc registered [ 6245.978380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6252.073898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6252.094372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.373378] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6253.387156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6253.471196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6256.635434] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6264.231563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6264.310393] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6264.329410] Lustre: 226005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6264.525212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6269.268240] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6276.280572] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6281.422114] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6286.868772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6287.038750] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6287.231167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6292.318348] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6294.162705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6298.670472] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6302.200241] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6308.679293] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6317.891696] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6327.245083] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6335.623438] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6355.222332] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6355.230244] Lustre: Skipped 1 previous similar message [ 6390.213146] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6390.221925] Lustre: Skipped 3 previous similar messages [ 6462.297278] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6462.303949] Lustre: Skipped 8 previous similar messages [ 6488.547704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6488.555107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6488.570775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6489.058194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.076353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6489.080052] Lustre: Skipped 1 previous similar message [ 6493.399925] Lustre: server umount lustre-OST0000 complete [ 6499.299588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6499.303858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6499.317104] Lustre: Skipped 1 previous similar message [ 6499.331678] Lustre: Skipped 1 previous similar message [ 6502.988614] Lustre: server umount lustre-MDT0000 complete [ 6506.897967] LustreError: 225999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170828 with bad export cookie 16331533933545041320 [ 6506.902768] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6506.917319] LustreError: 225999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6507.410438] Lustre: server umount lustre-MDT0001 complete [ 6517.860375] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6520.456484] Key type lgssc unregistered [ 6520.704084] LNet: 230837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6520.717247] LNetError: 230837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6520.756955] LNet: Removed LNI 192.168.206.140@tcp [ 6521.447332] Key type .llcrypt unregistered [ 6521.449927] Key type ._llcrypt unregistered [ 6544.907049] Key type ._llcrypt registered [ 6544.909403] Key type .llcrypt registered [ 6545.007927] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6546.296307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6546.321154] alg: No test for adler32 (adler32-zlib) [ 6547.364329] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 6547.619545] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 6549.296448] Key type lgssc registered [ 6550.274780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6557.104937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6557.125487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6558.560530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6562.781891] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6563.821048] LustreError: 232260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6563.841624] LustreError: 232260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6568.931511] LustreError: 232261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6571.577191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.914329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6576.150186] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6582.277910] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6587.566250] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6594.096445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6594.463228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6600.712180] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6603.704603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6608.290983] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6627.298867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6627.320230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6627.351979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6628.656950] Lustre: server umount lustre-OST0000 complete [ 6633.448930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6633.466060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6633.479557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6634.469531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.494887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6634.503708] Lustre: Skipped 1 previous similar message [ 6639.587106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6639.592429] Lustre: Skipped 1 previous similar message [ 6641.727805] Lustre: server umount lustre-MDT0000 complete [ 6644.707041] LustreError: 232261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6644.724424] LustreError: 232261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6645.099167] LustreError: 234841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170966 with bad export cookie 11265155361626412985 [ 6645.099827] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6645.114151] LustreError: 234841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6645.370970] Lustre: server umount lustre-MDT0001 complete [ 6654.717000] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6656.902525] Key type lgssc unregistered [ 6657.120851] LNet: 235955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6657.126803] LNetError: 235955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6658.156889] LNet: Removed LNI 192.168.206.140@tcp [ 6658.929884] Key type .llcrypt unregistered [ 6658.933236] Key type ._llcrypt unregistered [ 6680.530958] Key type ._llcrypt registered [ 6680.533534] Key type .llcrypt registered [ 6680.660556] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6681.827437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6681.853542] alg: No test for adler32 (adler32-zlib) [ 6682.911660] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 6683.152115] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 6684.808958] Key type lgssc registered [ 6685.796308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6958.595280] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6961.809810] Key type lgssc unregistered [ 6962.183596] LNet: 237979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6962.200805] LNetError: 237979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6962.220452] LNet: Removed LNI 192.168.206.140@tcp [ 6963.139924] Key type .llcrypt unregistered [ 6963.146347] Key type ._llcrypt unregistered [ 6988.845882] Key type ._llcrypt registered [ 6988.847596] Key type .llcrypt registered [ 6988.952225] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 7005.829674] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 7006.669120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7006.705769] alg: No test for adler32 (adler32-zlib) [ 7007.792898] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 7008.025560] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 7009.664195] Key type lgssc registered [ 7010.635619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7040.270378] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6785 sec ========== 06:15:59 (1788171359) [ 7042.006529] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:16:02 (1788171362) === [ 7045.562700] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:16:06 (1788171366) === [ 7087.620663] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 7090.458360] Key type lgssc unregistered [ 7090.713695] LNet: 243224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7090.728606] LNetError: 243224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7090.758558] LNet: Removed LNI 192.168.206.140@tcp [ 7091.806798] Key type .llcrypt unregistered [ 7091.809752] Key type ._llcrypt unregistered