[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509058219 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003240] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008527] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011139] LSM: Security Framework initializing [ 0.012039] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014068] *** VALIDATE selinux *** [ 0.023246] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027766] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.032340] *** VALIDATE proc *** [ 0.033248] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035279] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.041803] debug: unmapping init [mem 0xffffffff8c259000-0xffffffff8c260fff] [ 0.043183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044732] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046016] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052324] rcu: Hierarchical SRCU implementation. [ 0.054564] smp: Bringing up secondary CPUs ... [ 0.055602] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.059409] smp: Brought up 1 node, 4 CPUs [ 0.061014] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.135932] node 0 deferred pages initialised in 70ms [ 0.138536] devtmpfs: initialized [ 0.140202] x86/mm: Memory block size: 128MB [ 0.143647] gcov: version magic: 0x41383552 [ 0.146234] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149127] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151313] pinctrl core: initialized pinctrl subsystem [ 0.153180] [ 0.153700] ************************************************************* [ 0.156014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158013] ** ** [ 0.160014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163015] ** ** [ 0.165011] ** This means that this kernel is built to expose internal ** [ 0.167014] ** IOMMU data structures, which may compromise security on ** [ 0.169017] ** your system. ** [ 0.172019] ** ** [ 0.174012] ** If you see this message and you are not debugging the ** [ 0.176013] ** kernel, report this immediately to your vendor! ** [ 0.178015] ** ** [ 0.180013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182026] ************************************************************* [ 0.185474] NET: Registered protocol family 16 [ 0.187434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193274] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197458] cpuidle: using governor menu [ 0.198633] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201360] PCI: Using configuration type 1 for base access [ 0.203109] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.213132] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215056] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.219040] cryptd: max_cpu_qlen set to 1000 [ 0.222203] ACPI: Added _OSI(Module Device) [ 0.223013] ACPI: Added _OSI(Processor Device) [ 0.225014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.227013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232364] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.238307] ACPI: Interpreter enabled [ 0.240067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.242014] ACPI: Using IOAPIC for interrupt routing [ 0.244146] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247447] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258113] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.260039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.272423] acpiphp: Slot [2] registered [ 0.273117] acpiphp: Slot [5] registered [ 0.275118] acpiphp: Slot [6] registered [ 0.276130] acpiphp: Slot [7] registered [ 0.278108] acpiphp: Slot [8] registered [ 0.279112] acpiphp: Slot [9] registered [ 0.281146] acpiphp: Slot [10] registered [ 0.282125] acpiphp: Slot [3] registered [ 0.284131] acpiphp: Slot [4] registered [ 0.286102] acpiphp: Slot [11] registered [ 0.288130] acpiphp: Slot [12] registered [ 0.289102] acpiphp: Slot [13] registered [ 0.291173] acpiphp: Slot [14] registered [ 0.292089] acpiphp: Slot [15] registered [ 0.294121] acpiphp: Slot [16] registered [ 0.296167] acpiphp: Slot [17] registered [ 0.297100] acpiphp: Slot [18] registered [ 0.299130] acpiphp: Slot [19] registered [ 0.301169] acpiphp: Slot [20] registered [ 0.302087] acpiphp: Slot [21] registered [ 0.304149] acpiphp: Slot [22] registered [ 0.306155] acpiphp: Slot [23] registered [ 0.307102] acpiphp: Slot [24] registered [ 0.309117] acpiphp: Slot [25] registered [ 0.310092] acpiphp: Slot [26] registered [ 0.312104] acpiphp: Slot [27] registered [ 0.313122] acpiphp: Slot [28] registered [ 0.314103] acpiphp: Slot [29] registered [ 0.316108] acpiphp: Slot [30] registered [ 0.318120] acpiphp: Slot [31] registered [ 0.319091] PCI host bridge to bus 0000:00 [ 0.321022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.329024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.337592] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341010] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.344460] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.355010] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.360109] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.363019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.365017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.368020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370602] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373840] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379926] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.398017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408168] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.421015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.436017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.445313] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.463018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.479019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.488981] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.496013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.506017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.520022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.529785] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.537018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.547016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.563017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.572188] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.578016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.584018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.596013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.602026] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.612028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.617019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.629017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.638960] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.642492] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.644447] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.647367] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.649264] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.655063] iommu: Default domain type: Passthrough [ 0.657443] SCSI subsystem initialized [ 0.658120] ACPI: bus type USB registered [ 0.660096] usbcore: registered new interface driver usbfs [ 0.661070] usbcore: registered new interface driver hub [ 0.663083] usbcore: registered new device driver usb [ 0.665166] pps_core: LinuxPPS API ver. 1 registered [ 0.667009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.669074] PTP clock support registered [ 0.671078] EDAC MC: Ver: 3.0.0 [ 0.673130] PCI: Using ACPI for IRQ routing [ 0.674996] NetLabel: Initializing [ 0.677012] NetLabel: domain hash size = 128 [ 0.678010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.680092] NetLabel: unlabeled traffic allowed by default [ 0.683124] vgaarb: loaded [ 0.685270] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.686015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.692390] clocksource: Switched to clocksource kvm-clock [ 0.797837] VFS: Disk quotas dquot_6.6.0 [ 0.799421] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.801945] *** VALIDATE ramfs *** [ 0.803312] *** VALIDATE hugetlbfs *** [ 0.804766] pnp: PnP ACPI init [ 0.807165] pnp: PnP ACPI: found 6 devices [ 0.823136] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.825945] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.827948] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.829889] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.831918] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.834247] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.837071] NET: Registered protocol family 2 [ 0.839651] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.844373] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.847647] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.852315] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.855373] TCP: Hash tables configured (established 65536 bind 65536) [ 0.858049] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.860190] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.863139] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.865111] NET: Registered protocol family 1 [ 0.868681] RPC: Registered named UNIX socket transport module. [ 0.871047] RPC: Registered udp transport module. [ 0.872781] RPC: Registered tcp transport module. [ 0.875645] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.878084] NET: Registered protocol family 44 [ 0.879707] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.881519] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.883236] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.885341] PCI: CLS 0 bytes, default 64 [ 0.886891] Unpacking initramfs... [ 2.279941] debug: unmapping init [mem 0xffff9a09fcc54000-0xffff9a09fffbffff] [ 2.283837] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.286286] software IO TLB: mapped [mem 0x00000000a2000000-0x00000000a6000000] (64MB) [ 2.289340] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.780936] Initialise system trusted keyrings [ 2.782779] Key type blacklist registered [ 2.784769] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.793712] zbud: loaded [ 2.795802] *** VALIDATE nfs *** [ 2.796732] *** VALIDATE nfs4 *** [ 2.798265] pstore: using deflate compression [ 2.801851] Platform Keyring initialized [ 2.896184] NET: Registered protocol family 38 [ 2.897784] Key type asymmetric registered [ 2.899061] Asymmetric key parser 'x509' registered [ 2.900601] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.904641] io scheduler mq-deadline registered [ 2.906370] io scheduler kyber registered [ 2.907954] io scheduler bfq registered [ 2.909886] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.912346] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.914668] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.916835] ACPI: Power Button [PWRF] [ 2.922122] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.928858] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.941097] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.949717] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.982906] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.010546] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.037885] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.042370] Non-volatile memory driver v1.3 [ 3.043953] Linux agpgart interface v0.103 [ 3.073959] virtio_blk virtio1: [vda] 147232 512-byte logical blocks (75.4 MB/71.9 MiB) [ 3.076682] vda: detected capacity change from 0 to 75382784 [ 3.093742] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.095703] vdb: detected capacity change from 0 to 1073741824 [ 3.111075] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.114281] vdc: detected capacity change from 0 to 2621440000 [ 3.128091] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.131297] vdd: detected capacity change from 0 to 2621440000 [ 3.145874] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.148833] vde: detected capacity change from 0 to 4294967296 [ 3.164077] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.166296] vdf: detected capacity change from 0 to 4294967296 [ 3.171276] libphy: Fixed MDIO Bus: probed [ 3.183634] usbcore: registered new interface driver usbserial_generic [ 3.186018] usbserial: USB Serial support registered for generic [ 3.188068] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.192038] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.194965] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.197303] mousedev: PS/2 mouse device common for all mice [ 3.200167] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.202562] rtc_cmos 00:05: RTC can wake from S4 [ 3.206187] rtc_cmos 00:05: registered as rtc0 [ 3.206488] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.207989] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.214029] intel_pstate: CPU model not supported [ 3.217468] hid: raw HID events driver (C) Jiri Kosina [ 3.217478] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.219394] usbcore: registered new interface driver usbhid [ 3.224237] usbhid: USB HID core driver [ 3.225788] drop_monitor: Initializing network drop monitor service [ 3.227443] Initializing XFRM netlink socket [ 3.228911] NET: Registered protocol family 10 [ 3.232121] Segment Routing with IPv6 [ 3.232877] NET: Registered protocol family 17 [ 3.235187] mpls_gso: MPLS GSO support [ 3.239410] RAS: Correctable Errors collector initialized. [ 3.241051] AVX version of gcm_enc/dec engaged. [ 3.241959] AES CTR mode by8 optimization enabled [ 3.301897] sched_clock: Marking stable (3301878035, 0)->(4192836812, -890958777) [ 3.304327] registered taskstats version 1 [ 3.305715] Loading compiled-in X.509 certificates [ 3.307420] zswap: loaded using pool lzo/zbud [ 3.330147] Key type big_key registered [ 3.339133] Key type encrypted registered [ 3.340771] ima: No TPM chip found, activating TPM-bypass! [ 3.342845] ima: Allocated hash algorithm: sha1 [ 3.344532] ima: No architecture policies found [ 3.346178] evm: Initialising EVM extended attributes: [ 3.348183] evm: security.selinux [ 3.349432] evm: security.ima [ 3.350565] evm: security.capability [ 3.351876] evm: HMAC attrs: 0x1 [ 3.354325] rtc_cmos 00:05: setting system clock to 2026-09-11 10:45:31 UTC (1789123531) [ 3.360473] debug: unmapping init [mem 0xffffffff8d203000-0xffffffff8d3fffff] [ 3.363627] debug: unmapping init [mem 0xffffffff8bf82000-0xffffffff8c258fff] [ 3.373132] Write protecting the kernel read-only data: 28672k [ 3.376816] debug: unmapping init [mem 0xffffffff8a603000-0xffffffff8a7fffff] [ 3.379822] debug: unmapping init [mem 0xffffffff8af14000-0xffffffff8affffff] [ 3.415875] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.424686] systemd[1]: Detected virtualization kvm. [ 3.426650] systemd[1]: Detected architecture x86-64. [ 3.428473] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.451972] systemd[1]: No hostname configured. [ 3.454362] systemd[1]: Set hostname to . [ 3.457137] random: systemd: uninitialized urandom read (16 bytes read) [ 3.460667] systemd[1]: Initializing machine ID from random generator. [ 3.504903] random: ln: uninitialized urandom read (6 bytes read) [ 3.595274] random: systemd: uninitialized urandom read (16 bytes read) [ 3.597225] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.601431] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.605557] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.121242] device-mapper: uevent: version 1.0.3 [ 4.123219] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.804728] random: fast init done [ 4.821219] virtio_net virtio0 ens2: renamed from eth0 [ 4.922850] scsi host0: ata_piix [ 4.935895] scsi host1: ata_piix [ 4.937633] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.940060] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.592832] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.857537] random: crng init done [ 9.859537] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.220347] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.369227] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.628714] SELinux: Disabled at runtime. [ 11.698247] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.707717] systemd[1]: Detected virtualization kvm. [ 11.710019] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.242079] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.245499] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.251268] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.256465] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.259972] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.267745] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.285427] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root F[ 12.413725] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ile System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev[ 12.775834] squashfs: version 4.0 (2009/01/31) Phillip Lougher Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.123906] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.164071] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.341491] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.351829] EDAC sbridge: Ver: 1.1.2 [ 15.220505] Key type dns_resolver registered [ 15.529505] NFS: Registering the id_resolver key type [ 15.531353] Key type id_resolver registered [ 15.532981] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg123-server login: [ 52.958628] libcfs: loading out-of-tree module taints kernel. [ 52.993343] Key type ._llcrypt registered [ 52.995300] Key type .llcrypt registered [ 53.140862] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 76.801430] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 78.348467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.362328] alg: No test for adler32 (adler32-zlib) [ 80.116412] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 81.057033] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 82.871276] Key type lgssc registered [ 84.896551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 93.711673] hrtimer: interrupt took 3464415 ns [ 106.261427] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 152.569088] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 167.146912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 167.209637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 168.575203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 168.615881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 168.745658] Lustre: lustre-MDT0000: new disk, initializing [ 168.823494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.857361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 173.844610] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.416509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.579393] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 187.606746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 187.609703] Lustre: Skipped 1 previous similar message [ 187.685493] Lustre: lustre-MDT0001: new disk, initializing [ 187.773245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.820941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.839349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 193.200626] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.771584] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 210.168699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.487875] Lustre: lustre-OST0000: new disk, initializing [ 210.492652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.505941] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 210.590432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.586911] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.149385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 218.172132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.259107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 232.022614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.253218] Lustre: lustre-OST0001: new disk, initializing [ 232.260344] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 232.265861] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.383233] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 239.285572] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.749244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 239.756085] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 239.909293] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 253.195800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 260.910268] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 268.725131] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing check_logdir /tmp/testlogs/ [ 274.422469] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing yml_node [ 278.725365] Lustre: DEBUG MARKER: Client: 2.17.57.138 [ 281.664774] Lustre: DEBUG MARKER: MDS: 2.17.57.138 [ 284.548054] Lustre: DEBUG MARKER: OSS: 2.17.57.138 [ 286.135968] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 06:50:12 EDT 2026 [ 304.907658] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 306.540432] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 326.112104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 326.120218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.133436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.136382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.141476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.150454] Lustre: Skipped 1 previous similar message [ 327.163462] Lustre: Skipped 3 previous similar messages [ 329.215906] Lustre: server umount lustre-MDT0000 complete [ 333.534697] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789123861 with bad export cookie 15998989048221202805 [ 333.554039] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 333.560368] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 337.379692] LustreError: 6517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.381024] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.420829] LustreError: 6517:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 337.456505] Lustre: Skipped 1 previous similar message [ 337.465223] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 337.474445] Lustre: Skipped 1 previous similar message [ 338.401166] LustreError: 6516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.420157] LustreError: 6516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 340.034325] Lustre: server umount lustre-MDT0001 complete [ 344.360351] Lustre: server umount lustre-OST0000 complete [ 349.288481] Lustre: server umount lustre-OST0001 complete [ 357.979736] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 369.518036] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 415.808082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.254152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 416.340655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.432479] Lustre: lustre-MDT0000: new disk, initializing [ 416.559821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.583178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 422.710315] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 436.539857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 436.635206] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 436.673426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 436.676590] Lustre: Skipped 1 previous similar message [ 436.729891] Lustre: lustre-MDT0001: new disk, initializing [ 436.871465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 436.926597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 436.939563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.640878] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 454.610949] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 461.527688] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 469.214281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 469.757631] Lustre: lustre-OST0000: new disk, initializing [ 469.763207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 469.772259] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 469.873135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.240624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 471.255230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 471.423582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 476.547919] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 486.091239] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 491.571282] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 491.725766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 496.970668] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 497.225262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 502.243707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 502.247100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.275527] Lustre: Skipped 1 previous similar message [ 502.292502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.296245] Lustre: Skipped 2 previous similar messages [ 505.467332] Lustre: server umount lustre-OST0000 complete [ 512.489536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.493286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 512.508831] Lustre: Skipped 2 previous similar messages [ 516.058617] Lustre: server umount lustre-MDT0000 complete [ 517.609506] LustreError: 15083:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.634694] LustreError: 15083:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 520.492852] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124048 with bad export cookie 15998989048221204338 [ 520.494499] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.501864] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 521.028567] Lustre: server umount lustre-MDT0001 complete [ 524.758147] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 06:54:11 (1789124051) [ 534.224354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.956174] LustreError: 19312:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.078448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 540.134352] LustreError: 19312:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.155171] LustreError: 19312:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 540.388957] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 550.323033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.371130] LustreError: 19313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.395565] LustreError: 19313:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 555.997962] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 564.418598] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 572.064361] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 580.204413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 580.592506] LustreError: 21131:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.305662] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 597.310512] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 606.697462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 606.713109] LustreError: Skipped 1 previous similar message [ 606.725203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.744498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 606.753533] Lustre: Skipped 1 previous similar message [ 610.790620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 612.425259] Lustre: server umount lustre-OST0000 complete [ 617.440628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 623.504166] Lustre: server umount lustre-MDT0000 complete [ 626.145583] LustreError: 19313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.160954] LustreError: 19313:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 627.653837] LustreError: 19292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124155 with bad export cookie 15998989048221205220 [ 627.666821] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.673737] LustreError: 19292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 628.140401] Lustre: server umount lustre-MDT0001 complete [ 642.065986] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 645.387526] Key type lgssc unregistered [ 645.653657] LNet: 22847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.662238] LNetError: 22847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.676649] LNet: Removed LNI 192.168.201.123@tcp [ 646.817834] Key type .llcrypt unregistered [ 646.823185] Key type ._llcrypt unregistered [ 673.420666] Key type ._llcrypt registered [ 673.425942] Key type .llcrypt registered [ 673.706604] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 675.251296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 675.275770] alg: No test for adler32 (adler32-zlib) [ 676.524574] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 676.961527] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 678.759302] Key type lgssc registered [ 680.053684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 687.816902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 687.848381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.450393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 695.043799] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 699.874230] LustreError: 24262:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.893362] LustreError: 24262:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 704.854256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 705.015816] LustreError: 24261:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 705.409690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 710.179342] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 719.266950] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 725.770850] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 733.322494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 733.648395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 735.739388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 740.598689] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 750.566384] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 770.538803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.546651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 770.566047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 770.589105] Lustre: Skipped 1 previous similar message [ 775.656513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 775.663108] Lustre: Skipped 1 previous similar message [ 780.773172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 780.784275] Lustre: Skipped 1 previous similar message [ 784.354286] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 784.511865] Lustre: server umount lustre-OST0000 complete [ 791.009489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.011557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 791.023649] Lustre: Skipped 2 previous similar messages [ 791.039953] Lustre: Skipped 1 previous similar message [ 794.796738] Lustre: server umount lustre-MDT0000 complete [ 796.132739] LustreError: 25265:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 796.151664] LustreError: 25265:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 798.552846] LustreError: 24244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124326 with bad export cookie 10820561034917794034 [ 798.579390] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 798.581622] LustreError: 24244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 798.877660] Lustre: server umount lustre-MDT0001 complete [ 810.354733] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 813.663719] Key type lgssc unregistered [ 814.102537] LNet: 28243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.106375] LNetError: 28243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.129759] LNet: Removed LNI 192.168.201.123@tcp [ 815.110167] Key type .llcrypt unregistered [ 815.113993] Key type ._llcrypt unregistered [ 837.959727] Key type ._llcrypt registered [ 837.974988] Key type .llcrypt registered [ 838.089687] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 839.116433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 839.400640] alg: No test for adler32 (adler32-zlib) [ 840.412735] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 840.646767] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 842.303181] Key type lgssc registered [ 843.397593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 851.635107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 851.674382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 853.406969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.240176] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 863.722113] LustreError: 29656:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 863.741419] LustreError: 29656:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 868.835087] LustreError: 29657:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.174984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.826480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 874.831653] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 883.732189] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 890.855061] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 906.637867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 907.091104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.750978] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 923.084514] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 923.504309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 929.762708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 929.767051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.774772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.861167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.887341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.907903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 936.069307] Lustre: server umount lustre-OST0000 complete [ 941.537450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 941.544148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.555676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 941.563819] Lustre: Skipped 1 previous similar message [ 944.099536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 944.101966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.648937] Lustre: server umount lustre-MDT0000 complete [ 949.231132] LustreError: 29653:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.250029] LustreError: 29653:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 951.101522] LustreError: 29638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124479 with bad export cookie 10152852611496210778 [ 951.102852] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 951.110822] LustreError: 29638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 951.470662] Lustre: server umount lustre-MDT0001 complete [ 963.193725] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 965.635366] Key type lgssc unregistered [ 965.930114] LNet: 33552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.939164] LNetError: 33552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.024855] LNet: Removed LNI 192.168.201.123@tcp [ 968.017706] Key type .llcrypt unregistered [ 968.022269] Key type ._llcrypt unregistered [ 996.378903] Key type ._llcrypt registered [ 996.380889] Key type .llcrypt registered [ 996.451297] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 998.059561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.132188] alg: No test for adler32 (adler32-zlib) [ 999.504633] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 999.850712] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1001.631566] Key type lgssc registered [ 1002.847123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1012.264537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1012.286775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.987702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1019.484887] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1024.494249] LustreError: 34965:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.539195] LustreError: 34965:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1025.508262] LustreError: 34966:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.921483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.414952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1034.819459] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1044.282418] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1050.879605] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1059.083920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1059.498139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1061.576836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1067.674675] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1077.363080] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1087.456839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1087.468396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.477883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1091.554340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1091.557974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1091.559103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1091.562604] Lustre: Skipped 1 previous similar message [ 1093.017356] Lustre: server umount lustre-OST0000 complete [ 1101.283850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1101.292535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.300855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1104.473472] Lustre: server umount lustre-MDT0000 complete [ 1106.918955] LustreError: 35695:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.938488] LustreError: 35695:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1109.716390] LustreError: 34947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124637 with bad export cookie 11604454465180241620 [ 1109.723207] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1109.743864] LustreError: 34947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1110.331842] Lustre: server umount lustre-MDT0001 complete [ 1122.126518] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1125.418502] Key type lgssc unregistered [ 1125.752688] LNet: 38507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.761201] LNetError: 38507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.781506] LNet: Removed LNI 192.168.201.123@tcp [ 1126.722255] Key type .llcrypt unregistered [ 1126.724409] Key type ._llcrypt unregistered [ 1154.784662] Key type ._llcrypt registered [ 1154.786477] Key type .llcrypt registered [ 1154.926026] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1156.853663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1156.908000] alg: No test for adler32 (adler32-zlib) [ 1158.120824] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1158.559360] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1160.327219] Key type lgssc registered [ 1161.967361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1171.406989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1171.436628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1173.222792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1179.056748] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1183.731153] LustreError: 39925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.746347] LustreError: 39925:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1188.844095] LustreError: 39926:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.031840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1189.400774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1194.574678] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1203.173121] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1209.904435] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1217.372911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1217.589391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.629519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1226.256685] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1237.251519] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1244.128466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1244.144479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.157870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1244.168100] Lustre: Skipped 1 previous similar message [ 1249.252518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1249.256575] Lustre: Skipped 2 previous similar messages [ 1249.699686] Lustre: server umount lustre-OST0000 complete [ 1273.896224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.898809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1273.913175] Lustre: Skipped 2 previous similar messages [ 1278.208617] Lustre: server umount lustre-MDT0000 complete [ 1278.958043] LustreError: 39920:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1279.001529] LustreError: 39920:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1282.366658] LustreError: 39904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124810 with bad export cookie 3737661901517060828 [ 1282.378935] LustreError: 39904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1282.381255] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1289.219886] Lustre: server umount lustre-MDT0001 complete [ 1302.503096] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1306.419686] Key type lgssc unregistered [ 1306.762711] LNet: 43639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.772653] LNetError: 43639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.795160] LNet: Removed LNI 192.168.201.123@tcp [ 1307.746184] Key type .llcrypt unregistered [ 1307.751649] Key type ._llcrypt unregistered [ 1329.894506] Key type ._llcrypt registered [ 1329.899300] Key type .llcrypt registered [ 1329.989937] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1331.338676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1331.351292] alg: No test for adler32 (adler32-zlib) [ 1332.426975] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1332.713994] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1334.444282] Key type lgssc registered [ 1335.800410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1345.844852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1345.888569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.350207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.820696] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1357.807579] LustreError: 45033:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.837655] LustreError: 45033:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1362.915157] LustreError: 45032:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.074021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.670410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1370.132230] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1378.237981] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1385.304961] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1394.196742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1394.850559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.651808] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1412.713234] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1417.414196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1425.894425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1425.905419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.920389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1427.944036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1427.951489] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.433741] Lustre: server umount lustre-MDT0000 complete [ 1433.061643] LustreError: 46010:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.079289] LustreError: 46010:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1434.480837] LustreError: 45014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124962 with bad export cookie 5185271295747799094 [ 1434.482036] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1434.488528] LustreError: 45014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1438.176429] LustreError: 45029:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.177402] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.205927] LustreError: 45029:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1438.246424] Lustre: Skipped 1 previous similar message [ 1438.251349] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1438.261420] Lustre: Skipped 3 previous similar messages [ 1440.877235] Lustre: server umount lustre-MDT0001 complete [ 1463.771631] Lustre: server umount lustre-OST0000 complete [ 1482.630695] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1486.234428] Key type lgssc unregistered [ 1486.585659] LNet: 48901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.591073] LNetError: 48901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.610084] LNet: Removed LNI 192.168.201.123@tcp [ 1487.540986] Key type .llcrypt unregistered [ 1487.543439] Key type ._llcrypt unregistered [ 1515.631368] Key type ._llcrypt registered [ 1515.634899] Key type .llcrypt registered [ 1515.864615] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1517.497969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1517.567887] alg: No test for adler32 (adler32-zlib) [ 1519.072221] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1519.418742] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1521.255193] Key type lgssc registered [ 1523.344760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1535.060943] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1535.075414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1550.818240] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1555.939302] LustreError: 50291:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 1582.559292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.609191] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1597.700474] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1767.076826] Lustre: server umount lustre-OST0000 complete [ 1787.815300] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1791.550483] Key type lgssc unregistered [ 1791.811755] LNet: 51971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1791.816326] LNetError: 51971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1791.837907] LNet: Removed LNI 192.168.201.123@tcp [ 1792.833221] Key type .llcrypt unregistered [ 1792.834958] Key type ._llcrypt unregistered [ 1821.269940] Key type ._llcrypt registered [ 1821.275303] Key type .llcrypt registered [ 1821.526546] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1822.932549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1823.051294] alg: No test for adler32 (adler32-zlib) [ 1824.307298] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1824.781541] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1826.575249] Key type lgssc registered [ 1828.164749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1837.940719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1837.962467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1839.662050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1844.347664] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1849.826494] LustreError: 53388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1849.860072] LustreError: 53388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1853.918354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.253067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1859.165650] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1869.100311] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1877.387759] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1887.011825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1887.492565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.356705] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1904.664118] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1909.295031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1909.305529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.339019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1914.344891] Lustre: Skipped 1 previous similar message [ 1914.643099] Lustre: server umount lustre-OST0000 complete [ 1919.457881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.472566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1919.475840] Lustre: Skipped 1 previous similar message [ 1919.489848] Lustre: Skipped 1 previous similar message [ 1924.580146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1924.593015] Lustre: Skipped 2 previous similar messages [ 1925.771826] Lustre: server umount lustre-MDT0000 complete [ 1929.697048] LustreError: 53388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1929.718979] LustreError: 53388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1931.645287] LustreError: 53368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125459 with bad export cookie 10334530516876381188 [ 1931.650212] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1931.656500] LustreError: 53368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1932.412236] Lustre: server umount lustre-MDT0001 complete [ 1947.993895] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1951.550438] Key type lgssc unregistered [ 1952.068716] LNet: 56923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.082602] LNetError: 56923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.105876] LNet: Removed LNI 192.168.201.123@tcp [ 1953.365973] Key type .llcrypt unregistered [ 1953.368257] Key type ._llcrypt unregistered [ 1985.287479] Key type ._llcrypt registered [ 1985.289035] Key type .llcrypt registered [ 1985.395867] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1987.215371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1987.297621] alg: No test for adler32 (adler32-zlib) [ 1988.547509] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1988.899822] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1990.655496] Key type lgssc registered [ 1991.985795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2000.061736] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2000.077860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2015.648944] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2020.767439] LustreError: 58314:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 2047.523614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.983158] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2063.847161] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2072.310398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.018433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.150521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2078.190935] LustreError: 58340:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.224980] LustreError: 58340:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2080.271962] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2082.284633] LustreError: 59350:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.309460] LustreError: 59350:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2087.398594] LustreError: 59711:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.426186] LustreError: 59711:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2090.853262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2091.495256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2096.964988] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2106.621524] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2114.493889] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2117.100609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2117.101765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.122835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2117.132190] Lustre: Skipped 1 previous similar message [ 2117.144553] Lustre: Skipped 1 previous similar message [ 2122.223042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2122.227930] Lustre: Skipped 1 previous similar message [ 2123.299844] Lustre: server umount lustre-OST0000 complete [ 2146.857551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.868206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2146.876155] Lustre: Skipped 1 previous similar message [ 2146.893296] Lustre: Skipped 1 previous similar message [ 2153.442877] LustreError: 59355:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.460026] LustreError: 59355:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2153.497575] Lustre: server umount lustre-MDT0000 complete [ 2157.577248] LustreError: 58320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125685 with bad export cookie 3630797464022756064 [ 2157.584048] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2157.596881] LustreError: 58320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2164.182497] Lustre: server umount lustre-MDT0001 complete [ 2178.383440] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2181.979272] Key type lgssc unregistered [ 2182.417748] LNet: 62076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.426641] LNetError: 62076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.444266] LNet: Removed LNI 192.168.201.123@tcp [ 2183.351986] Key type .llcrypt unregistered [ 2183.354048] Key type ._llcrypt unregistered [ 2209.062885] Key type ._llcrypt registered [ 2209.064213] Key type .llcrypt registered [ 2209.170627] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2210.733479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2210.984620] alg: No test for adler32 (adler32-zlib) [ 2212.071973] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 2212.363087] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2214.065064] Key type lgssc registered [ 2215.347513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2224.514219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2224.537274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.912849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.757810] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2236.386091] LustreError: 63491:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.414610] LustreError: 63491:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2240.599557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.986099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2246.073772] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2258.098905] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2264.868238] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2272.873792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2273.516070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.078409] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2289.550941] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2296.052279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2311.662757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.671767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2311.680944] Lustre: Skipped 1 previous similar message [ 2311.711651] Lustre: Skipped 1 previous similar message [ 2314.175908] Lustre: server umount lustre-OST0000 complete [ 2321.890502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.899190] Lustre: Skipped 1 previous similar message [ 2321.899979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2326.182233] Lustre: server umount lustre-MDT0000 complete [ 2327.019279] LustreError: 63488:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.042788] LustreError: 63488:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2330.726416] LustreError: 63470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125858 with bad export cookie 5626795773352774836 [ 2330.734413] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2330.746973] LustreError: 63470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2331.207654] Lustre: server umount lustre-MDT0001 complete [ 2344.641696] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2347.707399] Key type lgssc unregistered [ 2348.088040] LNet: 67051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2348.097779] LNetError: 67051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2349.157900] LNet: Removed LNI 192.168.201.123@tcp [ 2350.405264] Key type .llcrypt unregistered [ 2350.407551] Key type ._llcrypt unregistered [ 2382.166743] Key type ._llcrypt registered [ 2382.168128] Key type .llcrypt registered [ 2382.297802] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2383.358243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2383.424835] alg: No test for adler32 (adler32-zlib) [ 2384.761710] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 2385.119882] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2386.848083] Key type lgssc registered [ 2387.659370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2397.080511] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 07:25:23 (1789125923) [ 2403.696282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2403.733460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.151348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.634862] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2415.587925] LustreError: 68754:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.599298] LustreError: 68754:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2419.051542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.504845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2423.870166] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2431.307394] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2438.304459] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2446.544054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.951404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.055238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2453.736765] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2463.193996] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2469.952901] Lustre: Failing over lustre-MDT0000 [ 2470.357901] Lustre: server umount lustre-MDT0000 complete [ 2470.881168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2470.889594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.909951] LustreError: 69481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.976444] LustreError: 68754:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.978874] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.986555] LustreError: 68754:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2480.107555] LustreError: 69481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2480.132351] LustreError: 69481:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 2481.971821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.081071] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2482.247380] Lustre: *** cfs_fail_loc=135, val=0*** [ 2482.377459] LustreError: 71841:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2482.389138] LustreError: 71841:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.123@tcp: cfg command failed: rc = -2 [ 2482.395504] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2482.395504] [ 2482.413190] LustreError: MGC192.168.201.123@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2482.425362] LustreError: 71830:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2482.430944] LustreError: 71830:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2482.437493] LustreError: 71830:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2482.442175] LustreError: 71830:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2482.461138] Lustre: server umount lustre-MDT0000 complete [ 2482.468180] LustreError: 71830:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2485.219815] LustreError: 68754:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.230434] LustreError: 68754:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2490.491536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.615127] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2490.631959] LustreError: 68748:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2490.893133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2490.943960] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2491.790733] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2495.776643] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2495.983378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2496.026053] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2496.096157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2505.703300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2505.710609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.724443] Lustre: Skipped 1 previous similar message [ 2505.736298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2509.943640] Lustre: server umount lustre-OST0000 complete [ 2516.449148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.452728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2516.468919] Lustre: Skipped 2 previous similar messages [ 2516.488800] Lustre: Skipped 3 previous similar messages [ 2520.286738] Lustre: server umount lustre-MDT0000 complete [ 2521.568349] LustreError: 68755:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.598084] LustreError: 68755:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 2524.751090] LustreError: 68734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126052 with bad export cookie 3560150488457738239 [ 2524.762131] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2524.771154] LustreError: 68734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2525.084624] Lustre: server umount lustre-MDT0001 complete [ 2538.053073] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2541.909671] Key type lgssc unregistered [ 2542.261049] LNet: 73629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2542.275503] LNetError: 73629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2542.303917] LNet: Removed LNI 192.168.201.123@tcp [ 2543.557553] Key type .llcrypt unregistered [ 2543.579948] Key type ._llcrypt unregistered [ 2567.255506] Key type ._llcrypt registered [ 2567.257401] Key type .llcrypt registered [ 2567.366738] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2569.107861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2569.137180] alg: No test for adler32 (adler32-zlib) [ 2570.176405] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 2570.416938] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2572.135375] Key type lgssc registered [ 2573.558076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2581.595108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2581.620227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.096591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2588.541161] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2593.252059] LustreError: 75034:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.270205] LustreError: 75034:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2597.269131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.624726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2602.095361] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2610.138929] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2616.108886] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2623.758610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2624.345508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.497833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2632.078184] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2641.106280] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2647.727406] Lustre: Failing over lustre-MDT0000 [ 2647.958931] Lustre: server umount lustre-MDT0000 complete [ 2648.039459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2648.049368] Lustre: Skipped 2 previous similar messages [ 2648.051031] LustreError: 75030:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.082282] LustreError: 75030:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2649.533245] LustreError: 75028:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.158793] LustreError: 75034:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.183081] LustreError: 75034:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2658.054861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.279448] LustreError: 75524:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.319831] LustreError: 75524:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2658.359780] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2658.658333] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2658.841137] LustreError: 78121:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2658.850947] LustreError: 78121:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.123@tcp: cfg command failed: rc = -9 [ 2658.857273] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2658.857273] [ 2658.863318] LustreError: MGC192.168.201.123@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2658.873405] LustreError: 78110:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2658.885328] LustreError: 78110:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2658.900858] LustreError: 78110:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2658.911670] LustreError: 78110:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2658.951883] Lustre: server umount lustre-MDT0000 complete [ 2658.961082] LustreError: 78110:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2666.269959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.537408] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.549813] LustreError: 75025:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2666.559577] LustreError: 78468:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.123@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000b1090b1d) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2666.958712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.012980] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2669.979978] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2671.729253] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2672.109790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2672.161846] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2672.212630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2682.343555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.347609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2682.367120] Lustre: Skipped 1 previous similar message [ 2682.387549] Lustre: Skipped 1 previous similar message [ 2686.884518] Lustre: server umount lustre-OST0000 complete [ 2692.576928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2692.581749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.593933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2692.622836] Lustre: Skipped 1 previous similar message [ 2692.651223] Lustre: Skipped 1 previous similar message [ 2697.708804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2702.842224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2702.851245] Lustre: Skipped 2 previous similar messages [ 2704.353134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2704.555726] Lustre: server umount lustre-MDT0000 complete [ 2707.937743] LustreError: 75030:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.957981] LustreError: 75030:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 2708.621789] LustreError: 75013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126236 with bad export cookie 14766210635798391002 [ 2708.630118] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2708.645353] LustreError: 75013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2709.123365] Lustre: server umount lustre-MDT0001 complete [ 2721.291248] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2724.424294] Key type lgssc unregistered [ 2724.792630] LNet: 79905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2724.811095] LNetError: 79905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2724.835745] LNet: Removed LNI 192.168.201.123@tcp [ 2725.867882] Key type .llcrypt unregistered [ 2725.870622] Key type ._llcrypt unregistered [ 2740.460640] Key type ._llcrypt registered [ 2740.462359] Key type .llcrypt registered [ 2740.565520] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2740.856819] Key type .llcrypt unregistered [ 2740.860422] Key type ._llcrypt unregistered [ 2763.687326] Key type ._llcrypt registered [ 2763.688800] Key type .llcrypt registered [ 2763.744734] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2764.876243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2764.935383] alg: No test for adler32 (adler32-zlib) [ 2766.122577] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 2766.238937] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2767.896347] Key type lgssc registered [ 2769.723545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2779.074701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2779.099760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.519194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.128922] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2789.883987] LustreError: 81822:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2789.917596] LustreError: 81822:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2794.982689] LustreError: 81823:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.633663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.153864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2803.685934] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2811.780001] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2818.145921] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2825.945916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2826.379729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.643453] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2841.456738] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2849.905124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2864.613583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2864.627529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2864.651292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2865.632730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2865.639231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2865.651225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2869.129535] Lustre: server umount lustre-OST0000 complete [ 2874.855769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2874.861586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2874.873398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2874.876288] Lustre: Skipped 1 previous similar message [ 2880.067458] Lustre: server umount lustre-MDT0000 complete [ 2880.996383] LustreError: 81823:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2881.030681] LustreError: 81823:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2883.755135] LustreError: 81803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126411 with bad export cookie 12291172199493697468 [ 2883.760328] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2883.764054] LustreError: 81803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2884.009411] Lustre: server umount lustre-MDT0001 complete [ 2901.257427] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2904.901985] Key type lgssc unregistered [ 2905.111177] LNet: 85383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2905.116698] LNetError: 85383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2906.156659] LNet: Removed LNI 192.168.201.123@tcp [ 2907.383554] Key type .llcrypt unregistered [ 2907.385581] Key type ._llcrypt unregistered [ 2937.801369] Key type ._llcrypt registered [ 2937.803752] Key type .llcrypt registered [ 2938.010242] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2939.739900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2939.986162] alg: No test for adler32 (adler32-zlib) [ 2941.138364] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 2941.449831] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2943.159167] Key type lgssc registered [ 2944.703799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2954.281910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2954.312390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.035583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.665039] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2966.516977] LustreError: 86797:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.549693] LustreError: 86797:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2971.618268] LustreError: 86796:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2972.192667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.019444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2978.625909] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2989.190111] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2997.359922] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3004.844943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3005.354583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.026097] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3020.678842] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3026.875496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 3032.036645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3032.044409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.046609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3032.065949] Lustre: Skipped 1 previous similar message [ 3032.099840] Lustre: Skipped 1 previous similar message [ 3034.423586] Lustre: server umount lustre-OST0000 complete [ 3039.712370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3039.735764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3039.758482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3042.275226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.276188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3042.318674] Lustre: Skipped 1 previous similar message [ 3045.391961] Lustre: server umount lustre-MDT0000 complete [ 3047.392822] LustreError: 86796:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.441562] LustreError: 86796:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3050.400506] LustreError: 86778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126578 with bad export cookie 17316183049948374185 [ 3050.404811] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3050.416639] LustreError: 86778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3050.873701] Lustre: server umount lustre-MDT0001 complete [ 3062.153882] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 3066.119949] Key type lgssc unregistered [ 3066.531870] LNet: 90335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3066.544085] LNetError: 90335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3066.561617] LNet: Removed LNI 192.168.201.123@tcp [ 3067.997080] Key type .llcrypt unregistered [ 3068.002513] Key type ._llcrypt unregistered [ 3092.588759] Key type ._llcrypt registered [ 3092.591149] Key type .llcrypt registered [ 3092.673763] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3093.915759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3093.950286] alg: No test for adler32 (adler32-zlib) [ 3095.000190] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 3095.314144] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 3097.008391] Key type lgssc registered [ 3098.085855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3107.296536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3107.336910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.870576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.614547] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3118.067063] LustreError: 91754:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.080711] LustreError: 91754:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3123.177806] LustreError: 91753:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.468849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.985736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3130.332570] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3139.450681] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3147.013768] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3154.878531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3155.317493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3162.950588] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3170.881391] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3178.894297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3193.832349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3193.858945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.870671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3195.315924] Lustre: server umount lustre-OST0000 complete [ 3199.457060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.470665] Lustre: Skipped 1 previous similar message [ 3199.472356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3199.486714] Lustre: Skipped 3 previous similar messages [ 3204.577198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3205.512424] Lustre: server umount lustre-MDT0000 complete [ 3209.184911] LustreError: 92487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126737 with bad export cookie 9799646426407868119 [ 3209.185736] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3209.194408] LustreError: 92487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3209.468649] Lustre: server umount lustre-MDT0001 complete [ 3224.029722] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 3229.235896] Key type lgssc unregistered [ 3229.674755] LNet: 95313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3229.685453] LNetError: 95313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3230.760669] LNet: Removed LNI 192.168.201.123@tcp [ 3232.217814] Key type .llcrypt unregistered [ 3232.219449] Key type ._llcrypt unregistered [ 3258.384555] Key type ._llcrypt registered [ 3258.385759] Key type .llcrypt registered [ 3258.441991] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3259.798855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3260.006612] alg: No test for adler32 (adler32-zlib) [ 3261.127383] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 3261.434353] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 3263.146129] Key type lgssc registered [ 3264.322262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3272.203193] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3272.221249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3287.839440] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3292.959436] LustreError: 96702:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 3319.648496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3327.571937] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3335.914244] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3342.862422] Lustre: server umount lustre-OST0000 complete [ 3351.874710] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 07:41:18 (1789126878) [ 3359.067296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.790576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.850451] LustreError: 98344:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3364.878417] LustreError: 98344:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3365.359415] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3368.942065] LustreError: 98345:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3374.063781] LustreError: 98344:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3375.192764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.673650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3381.087588] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3389.818408] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3397.155081] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3405.294424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3405.681225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.310849] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3419.029979] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3429.267698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3440.612696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3440.621973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.639331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3444.707765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3444.712095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3444.753922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3446.331167] Lustre: server umount lustre-OST0000 complete [ 3452.386868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3452.397556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3452.416787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3452.423240] Lustre: Skipped 1 previous similar message [ 3454.947619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.979244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3454.999557] Lustre: Skipped 1 previous similar message [ 3456.328913] Lustre: server umount lustre-MDT0000 complete [ 3460.073308] LustreError: 98345:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.106061] LustreError: 98345:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3461.436446] LustreError: 98325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126989 with bad export cookie 5517520077637082861 [ 3461.438683] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3461.450158] LustreError: 98325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3461.925295] Lustre: server umount lustre-MDT0001 complete [ 3480.519348] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 3484.313470] Key type lgssc unregistered [ 3484.641721] LNet: 102087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3484.646868] LNetError: 102087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3484.660281] LNet: Removed LNI 192.168.201.123@tcp [ 3485.600652] Key type .llcrypt unregistered [ 3485.604454] Key type ._llcrypt unregistered [ 3533.821220] Key type ._llcrypt registered [ 3533.823552] Key type .llcrypt registered [ 3533.911900] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3535.058747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3535.172870] alg: No test for adler32 (adler32-zlib) [ 3536.363821] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 3536.692225] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 3538.343198] Key type lgssc registered [ 3539.394715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3547.912199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3547.927265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.416954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3554.937638] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3558.883268] LustreError: 104276:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3558.893786] LustreError: 104276:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3563.758722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.070986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3569.242434] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3576.327818] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3583.238736] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3591.684290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3592.314503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3594.386329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3600.250459] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3609.075954] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3620.322584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.352557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3620.359615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3620.376355] Lustre: Skipped 1 previous similar message [ 3624.780036] Lustre: server umount lustre-OST0000 complete [ 3630.568117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3630.569819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3630.588677] Lustre: Skipped 1 previous similar message [ 3630.590298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3630.609678] Lustre: Skipped 1 previous similar message [ 3635.683917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3635.693304] Lustre: Skipped 1 previous similar message [ 3640.802063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3640.818729] Lustre: Skipped 1 previous similar message [ 3643.359151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3643.794955] Lustre: server umount lustre-MDT0000 complete [ 3645.920975] LustreError: 104277:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.958794] LustreError: 104277:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3648.983048] LustreError: 104258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127177 with bad export cookie 7392961146761567396 [ 3648.989997] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3649.009554] LustreError: 104258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3649.401416] Lustre: server umount lustre-MDT0001 complete [ 3662.086264] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 3665.770985] Key type lgssc unregistered [ 3666.106692] LNet: 107814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3666.115996] LNetError: 107814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3666.128980] LNet: Removed LNI 192.168.201.123@tcp [ 3667.006474] Key type .llcrypt unregistered [ 3667.007765] Key type ._llcrypt unregistered [ 3687.672741] Key type ._llcrypt registered [ 3687.677583] Key type .llcrypt registered [ 3687.860356] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3689.151168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3689.296057] alg: No test for adler32 (adler32-zlib) [ 3690.647305] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 3691.011856] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 3692.672373] Key type lgssc registered [ 3693.724613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3705.803070] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3705.849320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3721.631528] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3726.751509] LustreError: 109061:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 3753.440827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3760.361814] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3768.527552] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3775.754421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3776.003604] LustreError: MGC192.168.201.123@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3776.022045] LustreError: 110086:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3776.027509] LustreError: 110086:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3776.037314] LustreError: 110086:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3776.098517] Lustre: server umount lustre-MDT0000 complete [ 3776.099931] LustreError: 110086:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3794.534326] Lustre: server umount lustre-OST0000 complete [ 3806.009981] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 3819.425133] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3865.355948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3865.709395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3865.751866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3865.851464] Lustre: lustre-MDT0000: new disk, initializing [ 3865.910530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3865.924688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3871.896323] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3885.651585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3885.727724] Lustre: 112878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3885.751912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3885.753543] Lustre: Skipped 1 previous similar message [ 3885.790637] Lustre: lustre-MDT0001: new disk, initializing [ 3885.837206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3885.881504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3885.899682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3892.059650] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3905.631374] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3913.206267] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3921.827689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3922.257903] Lustre: lustre-OST0000: new disk, initializing [ 3922.263545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3922.270056] Lustre: 114849:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3922.442828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3929.194562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3929.209710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3929.343375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3930.790616] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3943.681859] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3952.456146] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3953.057186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3958.780840] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3959.225629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3965.410576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3965.418374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3965.420345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3965.430241] Lustre: Skipped 1 previous similar message [ 3970.532831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3970.545118] Lustre: Skipped 2 previous similar messages [ 3975.655822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3975.670871] Lustre: Skipped 1 previous similar message [ 3976.162637] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3976.374968] Lustre: server umount lustre-OST0000 complete [ 3983.329235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3983.334771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.340380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3985.891280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.256563] Lustre: server umount lustre-MDT0000 complete [ 3994.556097] LustreError: 112872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127522 with bad export cookie 11447387260806959021 [ 3994.580997] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3994.585674] LustreError: 112872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3994.945800] Lustre: server umount lustre-MDT0001 complete [ 4009.535984] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 07:52:15 (1789127535) [ 4013.994937] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4033.891783] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 4046.596816] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 4094.142149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.531948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4094.569967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4094.647473] Lustre: lustre-MDT0000: new disk, initializing [ 4094.750084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.763677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4100.241523] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4114.525265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4114.746534] Lustre: 119759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4114.868213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4114.880076] Lustre: Skipped 1 previous similar message [ 4114.978314] Lustre: lustre-MDT0001: new disk, initializing [ 4115.078268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4115.119870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4115.137088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4121.069865] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4133.032837] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4140.322611] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4150.338309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4150.891279] Lustre: lustre-OST0000: new disk, initializing [ 4150.898867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4150.911457] Lustre: 121731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4151.058644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.035116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4158.041473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4158.170873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4159.411853] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4171.720632] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4178.795224] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4179.274326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4184.485610] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4184.857216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4189.154038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.160676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4189.184750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4189.215761] Lustre: Skipped 2 previous similar messages [ 4193.951115] Lustre: server umount lustre-OST0000 complete [ 4199.395443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4199.407117] Lustre: Skipped 2 previous similar messages [ 4199.415869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4199.424624] Lustre: Skipped 2 previous similar messages [ 4204.526283] LustreError: 119771:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4204.538646] LustreError: 119771:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4204.798448] Lustre: server umount lustre-MDT0000 complete [ 4209.633692] LustreError: 119772:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4209.702400] LustreError: 119772:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4209.816626] LustreError: 119752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127737 with bad export cookie 11447387260806959903 [ 4209.839885] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4209.840894] LustreError: 119752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4210.455963] Lustre: server umount lustre-MDT0001 complete [ 4221.677330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4222.555276] LustreError: 123947:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4222.692408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4228.086060] LustreError: 123948:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4228.450830] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4232.231250] LustreError: 123947:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4238.672428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4245.253547] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4255.885828] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4264.646746] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4272.972769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4281.239732] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4291.761938] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4303.517043] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4308.454608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.456561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4308.483284] Lustre: Skipped 1 previous similar message [ 4314.785022] Lustre: server umount lustre-OST0000 complete [ 4321.256925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4334.047162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4336.229435] Lustre: server umount lustre-MDT0000 complete [ 4338.148073] LustreError: 125509:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4338.179637] LustreError: 125509:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4341.257589] LustreError: 123927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127869 with bad export cookie 11447387260806960785 [ 4341.262151] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4341.276030] LustreError: 123927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4341.848882] Lustre: server umount lustre-MDT0001 complete [ 4360.917983] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 4366.271597] Key type lgssc unregistered [ 4366.793798] LNet: 127598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4366.800550] LNetError: 127598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4366.809954] LNet: Removed LNI 192.168.201.123@tcp [ 4367.462829] Key type .llcrypt unregistered [ 4367.465096] Key type ._llcrypt unregistered [ 4390.581900] Key type ._llcrypt registered [ 4390.583702] Key type .llcrypt registered [ 4390.693075] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 4414.561172] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 4416.692763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4416.733622] alg: No test for adler32 (adler32-zlib) [ 4418.407034] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 4418.737443] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 4420.474875] Key type lgssc registered [ 4422.666396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4470.469326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4470.550035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.935902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4472.013558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4472.165674] Lustre: lustre-MDT0000: new disk, initializing [ 4472.368304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.397963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4478.922155] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4491.345738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.536971] Lustre: 130772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4491.612201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4491.623296] Lustre: Skipped 1 previous similar message [ 4491.765552] Lustre: lustre-MDT0001: new disk, initializing [ 4491.939960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4491.998741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4492.021247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4497.555526] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4510.473715] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4518.857349] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4527.733668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4528.152300] Lustre: lustre-OST0000: new disk, initializing [ 4528.167633] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4528.184829] Lustre: 132746:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4528.341858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4536.976224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4537.000150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4537.187600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4537.333613] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4550.608477] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4557.300165] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4557.656867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4564.330416] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4564.572260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4568.043256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.055046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4568.066841] Lustre: Skipped 1 previous similar message [ 4568.099739] Lustre: Skipped 1 previous similar message [ 4573.153557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4573.168130] Lustre: Skipped 1 previous similar message [ 4578.278372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4578.287275] Lustre: Skipped 1 previous similar message [ 4581.344088] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4581.832967] Lustre: server umount lustre-OST0000 complete [ 4588.519340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.525350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4588.537650] Lustre: Skipped 1 previous similar message [ 4588.553622] Lustre: Skipped 1 previous similar message [ 4592.742884] Lustre: server umount lustre-MDT0000 complete [ 4597.207219] LustreError: 130766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128125 with bad export cookie 4993286517632667900 [ 4597.213458] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4597.227330] LustreError: 130766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4597.842333] Lustre: server umount lustre-MDT0001 complete [ 4608.288770] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 08:02:14 (1789128134) [ 4616.918498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.565340] LustreError: 135170:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4617.586280] LustreError: 135170:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4617.668094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.361574] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4622.816656] LustreError: 135171:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.914942] LustreError: 135170:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4632.034064] LustreError: 135171:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.543223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4634.358027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4639.893048] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4649.773145] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4658.087749] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4664.299540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4664.321154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4667.397752] Lustre: server umount lustre-MDT0000 complete [ 4669.411681] LustreError: 135166:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4669.433912] LustreError: 135166:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4672.305953] LustreError: 135152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128200 with bad export cookie 4993286517632668782 [ 4672.331407] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4672.852995] Lustre: server umount lustre-MDT0001 complete [ 4684.369647] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 08:03:30 (1789128210) [ 4694.007136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4709.550503] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4714.656336] LustreError: 137585:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 4740.319296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4747.390766] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4755.797297] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4757.966885] Lustre: server umount lustre-OST0000 complete [ 4766.189634] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 08:04:52 (1789128292) [ 4773.157410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.565076] LustreError: 139033:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.644840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.913335] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4788.528215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.431599] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4804.240156] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4811.550707] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4818.928335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.331768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4819.336853] Lustre: Skipped 1 previous similar message [ 4826.003737] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4835.406902] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4854.245029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4854.255256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4854.262111] Lustre: Skipped 2 previous similar messages [ 4854.296899] Lustre: Skipped 3 previous similar messages [ 4860.419194] Lustre: server umount lustre-MDT0000 complete [ 4864.482628] LustreError: 140013:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4864.503133] LustreError: 140013:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 4864.558698] LustreError: 139014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128392 with bad export cookie 4993286517632669447 [ 4864.577625] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.579547] LustreError: 139014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4865.033967] Lustre: server umount lustre-MDT0001 complete [ 4879.162137] Lustre: server umount lustre-OST0000 complete [ 4892.723064] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 08:06:58 (1789128418) [ 4901.636575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4902.026939] LustreError: 142712:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4902.106078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4906.932527] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4918.319973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4923.719473] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4932.301518] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4938.884395] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4947.170312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4948.631651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4955.133929] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4964.170347] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4970.449841] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4970.680770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4976.794941] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4977.190860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4979.683488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4979.702842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4979.721912] Lustre: Skipped 1 previous similar message [ 4979.739797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4979.752867] Lustre: Skipped 3 previous similar messages [ 4985.634537] Lustre: server umount lustre-OST0000 complete [ 4990.444519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4990.451474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.466111] Lustre: Skipped 1 previous similar message [ 4996.572451] Lustre: server umount lustre-MDT0000 complete [ 4997.090249] LustreError: 143717:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4997.111154] LustreError: 143717:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 5001.148870] LustreError: 142693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128529 with bad export cookie 4993286517632670518 [ 5001.154522] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5001.164268] LustreError: 142693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5001.538321] Lustre: server umount lustre-MDT0001 complete [ 5013.037908] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 08:08:59 (1789128539) [ 5021.548675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5037.344566] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5042.463484] LustreError: 146872:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 5068.193309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5068.205178] Lustre: Skipped 2 previous similar messages [ 5075.527492] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5085.165764] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5093.389740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5094.137107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5099.781587] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5109.215819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5114.268283] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5122.179475] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5128.704061] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5133.934330] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5134.422216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5139.271440] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5139.515940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5145.570569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5145.576573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5145.586417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5145.594462] Lustre: Skipped 3 previous similar messages [ 5145.611602] Lustre: Skipped 7 previous similar messages [ 5156.319159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5156.671813] Lustre: server umount lustre-MDT0000 complete [ 5160.615469] LustreError: 146879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128688 with bad export cookie 4993286517632671540 [ 5160.618984] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5160.630671] LustreError: 146879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5160.942708] Lustre: server umount lustre-MDT0001 complete [ 5174.219830] Lustre: server umount lustre-OST0000 complete [ 5184.774811] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 08:11:51 (1789128711) [ 5192.454125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5208.032140] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5213.151704] LustreError: 151072:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 5244.949505] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5253.674948] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5261.669401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5262.011297] LustreError: 151097:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5262.035670] LustreError: 151097:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 5262.214085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5267.196916] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5276.011284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5281.855922] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5292.035865] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5297.618841] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5305.839423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5306.225793] Lustre: lustre-OST0001: new disk, initializing [ 5306.234038] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5306.247771] Lustre: 153944:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5312.701934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5312.712546] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5312.917108] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5315.085737] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5325.553160] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5332.342868] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5332.652757] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5338.805264] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5339.098690] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5343.213560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.232416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.260755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5343.275921] Lustre: Skipped 6 previous similar messages [ 5347.383192] Lustre: server umount lustre-OST0000 complete [ 5353.956298] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5353.968865] LustreError: Skipped 1 previous similar message [ 5367.265743] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5367.435934] Lustre: server umount lustre-OST0001 complete [ 5373.923166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5378.637747] Lustre: server umount lustre-MDT0000 complete [ 5383.244981] LustreError: 151080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128911 with bad export cookie 4993286517632672492 [ 5383.263519] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5383.270418] LustreError: 151080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5383.775936] Lustre: server umount lustre-MDT0001 complete [ 5410.827569] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 08:15:37 (1789128937) [ 5412.630266] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5414.511539] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5416.238877] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 08:15:43 (1789128943) [ 5424.065371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5424.497585] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5424.542903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5424.760011] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5424.763852] Lustre: Skipped 6 previous similar messages [ 5428.903282] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5442.250714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5442.531401] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5442.609644] Lustre: 157510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5449.037079] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5457.167687] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5463.274880] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5473.356632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5473.718245] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5482.237407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5482.467672] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5491.533670] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5499.329422] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5499.832468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5504.526967] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5504.692982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5508.073425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5508.076454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.092987] LustreError: Skipped 1 previous similar message [ 5508.097492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5508.106270] Lustre: Skipped 6 previous similar messages [ 5508.131053] Lustre: Skipped 11 previous similar messages [ 5513.898398] Lustre: server umount lustre-OST0000 complete [ 5540.531731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5550.174926] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5560.976866] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5563.824892] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.837923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.841836] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5563.874818] Lustre: Skipped 1 previous similar message [ 5567.980919] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5568.194640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5574.655202] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5575.098213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5583.735397] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff888dc5c2c000.ost_server_uuid 50 [ 5586.555569] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff888dc5c2c000.ost_server_uuid in IDLE state after 0 sec [ 5586.859465] LustreError: 157518:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.23@tcp) returned error from blocking AST (req@00000000658be082 x1876036625397632 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000096e8ef5e/0x454bb7a1d93e2c46 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.23@tcp remote: 0x92aaeba88463cb85 expref: 5 pid: 157517 timeout: 5686 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5586.932866] LustreError: lustre-MDT0000: A client on nid 192.168.201.23@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5586.956281] LustreError: 157507:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.23@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000096e8ef5e/0x454bb7a1d93e2c46 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.23@tcp remote: 0x92aaeba88463cb85 expref: 6 pid: 157517 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5594.592966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5600.340761] Lustre: server umount lustre-OST0000 complete [ 5611.958254] Lustre: server umount lustre-MDT0000 complete [ 5615.073262] LustreError: 157523:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.091391] LustreError: 157523:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [ 5616.424579] LustreError: 157503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129144 with bad export cookie 4993286517632673675 [ 5616.446539] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5623.084209] Lustre: server umount lustre-MDT0001 complete [ 5637.668886] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5641.272425] Key type lgssc unregistered [ 5641.614832] LNet: 163633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5641.631305] LNetError: 163633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5641.652071] LNet: Removed LNI 192.168.201.123@tcp [ 5642.971505] Key type .llcrypt unregistered [ 5642.974315] Key type ._llcrypt unregistered [ 5671.425520] Key type ._llcrypt registered [ 5671.428399] Key type .llcrypt registered [ 5671.596703] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5673.443211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5673.533232] alg: No test for adler32 (adler32-zlib) [ 5674.880750] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 5675.236857] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 5677.039352] Key type lgssc registered [ 5678.956820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5689.043039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5689.057943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5690.414193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.355766] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5700.577799] LustreError: 165049:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5700.608639] LustreError: 165049:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5704.615305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5705.397375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5711.071445] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5723.148619] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5730.066420] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5738.097100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5738.501828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5747.153541] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5756.302656] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5760.076974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5767.431899] Lustre: Failing over lustre-MDT0000 [ 5767.601476] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.23@tcp (stopping) [ 5767.802272] Lustre: server umount lustre-MDT0000 complete [ 5770.208675] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.210572] LustreError: 165044:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.226295] Lustre: Skipped 2 previous similar messages [ 5770.254273] LustreError: 165044:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5775.330707] LustreError: 165050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.362826] LustreError: 165050:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5780.450481] LustreError: 165050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5780.480947] LustreError: 165050:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5780.959526] LustreError: 165029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.23@tcp arrived at 1789129309 with bad export cookie 878372149774640223 [ 5780.979628] LustreError: 165029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5785.570606] LustreError: 165044:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5785.599690] LustreError: 165044:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5786.591872] Lustre: 164360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789129298/real 1789129298] req@00000000b25ac3df x1876037943194368/t0(0) o400->MGC192.168.201.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789129314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5786.641622] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5791.666026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5796.833596] LustreError: 165050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5796.869050] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc309ad0ffe9e5a1 [ 5796.873963] LustreError: 165050:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 5796.895055] Lustre: MGC192.168.201.123@tcp: Connection restored to 0@lo (at 0@lo) [ 5797.403383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5797.516146] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5802.468740] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5802.485051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5803.262693] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5806.162146] Lustre: lustre-MDT0000: Denying connection for new client e8525bc5-bde1-4c64-9482-6d63c9fe6051 (at 192.168.201.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5811.604661] Lustre: lustre-MDT0000: Denying connection for new client e8525bc5-bde1-4c64-9482-6d63c9fe6051 (at 192.168.201.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5817.312522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5817.317162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5817.327534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5821.507594] Lustre: server umount lustre-OST0000 complete [ 5826.544568] LustreError: 168787:0:(ldlm_lib.c:3067:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5826.560770] Lustre: 168136:0:(ldlm_lib.c:2467:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5826.570402] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5826.584240] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5826.600330] LustreError: 168136:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bbb1c6bb x1876037943215616/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5826.617311] LustreError: 168136:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5826.628398] LustreError: 168136:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5826.644564] Lustre: lustre-MDT0000: Recovery over after 0:24, of 2 clients 0 recovered and 2 were evicted. [ 5827.145475] Lustre: server umount lustre-MDT0000 complete [ 5828.072515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5828.079657] Lustre: Skipped 1 previous similar message [ 5828.086974] LustreError: 166528:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5831.633795] LustreError: 165030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129359 with bad export cookie 878372149774640545 [ 5831.656923] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5832.318802] Lustre: server umount lustre-MDT0001 complete [ 5845.195784] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5848.398923] Key type lgssc unregistered [ 5848.828943] LNet: 169545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5848.836361] LNetError: 169545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5848.858642] LNet: Removed LNI 192.168.201.123@tcp [ 5849.947867] Key type .llcrypt unregistered [ 5849.950643] Key type ._llcrypt unregistered [ 5878.557244] Key type ._llcrypt registered [ 5878.561680] Key type .llcrypt registered [ 5878.740637] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5879.941199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5880.101067] alg: No test for adler32 (adler32-zlib) [ 5881.196419] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 5881.402273] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 5883.152862] Key type lgssc registered [ 5884.318440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5893.918361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5893.969351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5895.794079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5901.333775] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5905.388159] LustreError: 170966:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5905.401738] LustreError: 170966:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5910.500294] LustreError: 170965:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5911.872393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5912.324648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5918.085164] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5927.758844] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5935.514330] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5943.847795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5944.432927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.074615] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5964.196807] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5964.946488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5975.527835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.539131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5975.549416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5975.564986] Lustre: Skipped 1 previous similar message [ 5975.603862] Lustre: Skipped 1 previous similar message [ 5980.709584] Lustre: server umount lustre-OST0000 complete [ 5989.344973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5989.350677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5989.364901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5990.883770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5990.886360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.903632] Lustre: server umount lustre-MDT0000 complete [ 5995.040835] LustreError: 171695:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5999.043203] LustreError: 170945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129527 with bad export cookie 5784960525690012308 [ 5999.061380] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5999.666549] Lustre: server umount lustre-MDT0001 complete [ 6013.710695] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 6016.957737] Key type lgssc unregistered [ 6017.541566] LNet: 174504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6017.547375] LNetError: 174504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6017.581336] LNet: Removed LNI 192.168.201.123@tcp [ 6018.292119] Key type .llcrypt unregistered [ 6018.294344] Key type ._llcrypt unregistered [ 6054.336888] Key type ._llcrypt registered [ 6054.343729] Key type .llcrypt registered [ 6054.498529] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 6056.037278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6056.089945] alg: No test for adler32 (adler32-zlib) [ 6057.204812] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 6057.412569] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 6059.216323] Key type lgssc registered [ 6060.750655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6071.391810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6071.441590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6073.176932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6079.506297] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6083.555720] LustreError: 176210:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6083.579350] LustreError: 176210:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6088.678439] LustreError: 176211:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6091.029234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6091.309772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6096.357467] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6105.719490] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6113.935588] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6122.758508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6123.199485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6133.218324] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6143.995601] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6144.722764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6160.357311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.366033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6160.374197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6160.768540] Lustre: server umount lustre-OST0000 complete [ 6168.040157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6168.052948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.060811] Lustre: Skipped 1 previous similar message [ 6168.068303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6168.072489] Lustre: Skipped 1 previous similar message [ 6170.592711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6170.599713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6170.637391] Lustre: Skipped 1 previous similar message [ 6172.302474] Lustre: server umount lustre-MDT0000 complete [ 6175.713259] LustreError: 176210:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6175.754566] LustreError: 176210:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6178.631380] LustreError: 176192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129706 with bad export cookie 4768492294195487800 [ 6178.649689] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6179.259761] Lustre: server umount lustre-MDT0001 complete [ 6194.012405] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 6197.443592] Key type lgssc unregistered [ 6197.686759] LNet: 179774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6197.695974] LNetError: 179774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6197.706664] LNet: Removed LNI 192.168.201.123@tcp [ 6198.584231] Key type .llcrypt unregistered [ 6198.586500] Key type ._llcrypt unregistered [ 6226.949728] Key type ._llcrypt registered [ 6226.951943] Key type .llcrypt registered [ 6227.142053] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 6228.958174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6228.993882] alg: No test for adler32 (adler32-zlib) [ 6230.404174] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 6230.777031] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 6232.495218] Key type lgssc registered [ 6233.928707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6247.243837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6247.282191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6248.726843] Lustre: *** cfs_fail_loc=135, val=0*** [ 6248.785982] LustreError: 181296:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6248.793163] LustreError: 181296:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.123@tcp: cfg command failed: rc = -2 [ 6248.806851] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6248.806851] [ 6248.816718] LustreError: MGC192.168.201.123@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6248.829118] LustreError: 181258:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6248.836518] LustreError: 181258:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6248.851724] LustreError: 181258:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6248.856768] LustreError: 181258:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6248.970774] Lustre: server umount lustre-MDT0000 complete [ 6248.974976] LustreError: 181258:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6257.947914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6258.618935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6264.633812] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6267.879800] LustreError: 181668:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6267.897459] LustreError: 181668:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6272.993653] LustreError: 181669:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6274.517964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.007514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6281.773994] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6293.039301] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6299.692886] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6305.763743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6305.766939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6305.792347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6305.803718] Lustre: Skipped 1 previous similar message [ 6307.858830] Lustre: server umount lustre-MDT0000 complete [ 6310.880196] LustreError: 181668:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.891086] LustreError: 181668:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6312.197875] LustreError: 182402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129840 with bad export cookie 5235773102246271931 [ 6312.208833] LustreError: 182402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6312.209902] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6312.531744] Lustre: server umount lustre-MDT0001 complete [ 6326.627491] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 6329.613857] Key type lgssc unregistered [ 6329.907722] LNet: 183994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6329.914312] LNetError: 183994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6329.936797] LNet: Removed LNI 192.168.201.123@tcp [ 6330.793147] Key type .llcrypt unregistered [ 6330.796958] Key type ._llcrypt unregistered [ 6364.152666] Key type ._llcrypt registered [ 6364.155380] Key type .llcrypt registered [ 6364.348452] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 6365.079042] Key type .llcrypt unregistered [ 6365.082228] Key type ._llcrypt unregistered [ 6388.819174] Key type ._llcrypt registered [ 6388.821958] Key type .llcrypt registered [ 6388.987138] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 6391.011436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6391.056415] alg: No test for adler32 (adler32-zlib) [ 6392.131595] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 6392.602987] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 6394.303168] Key type lgssc registered [ 6395.651889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6405.454935] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6405.478312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6421.087429] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6426.207266] LustreError: 186136:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 6452.959229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6461.401660] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6470.652774] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6479.247819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.891387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6479.980926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6484.968471] LustreError: 186162:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6484.998519] LustreError: 186162:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6485.127645] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6487.018666] LustreError: 186161:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.046131] LustreError: 186161:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6492.156930] LustreError: 186162:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6492.196720] LustreError: 186162:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6495.855460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.277684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6503.752200] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6513.072475] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6520.815142] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6525.006408] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6537.186382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6537.197496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6537.205342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6537.212322] Lustre: Skipped 1 previous similar message [ 6537.229573] Lustre: Skipped 1 previous similar message [ 6542.311110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6542.328144] Lustre: Skipped 1 previous similar message [ 6542.722759] Lustre: server umount lustre-OST0000 complete [ 6552.546185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6552.561416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.598093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6552.645532] Lustre: Skipped 1 previous similar message [ 6557.665246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6557.685179] Lustre: Skipped 2 previous similar messages [ 6562.276296] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6562.510379] Lustre: server umount lustre-MDT0000 complete [ 6562.794299] LustreError: 187178:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6562.811588] LustreError: 187178:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6566.887216] LustreError: 187179:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6566.899900] LustreError: 187179:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6567.193158] LustreError: 188799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130095 with bad export cookie 16415464311588031945 [ 6567.194819] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6567.205084] LustreError: 188799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6567.868321] Lustre: server umount lustre-MDT0001 complete [ 6581.004650] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 6584.680548] Key type lgssc unregistered [ 6585.020488] LNet: 190107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6585.029438] LNetError: 190107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6585.047930] LNet: Removed LNI 192.168.201.123@tcp [ 6586.136817] Key type .llcrypt unregistered [ 6586.142140] Key type ._llcrypt unregistered [ 6620.375107] Key type ._llcrypt registered [ 6620.381306] Key type .llcrypt registered [ 6620.608719] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 6622.826449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6622.904157] alg: No test for adler32 (adler32-zlib) [ 6624.149982] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 6624.492818] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 6626.159536] Key type lgssc registered [ 6628.048759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6637.628164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6637.678102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.420826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6646.608862] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6649.824865] LustreError: 191521:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6649.851073] LustreError: 191521:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6654.950115] LustreError: 191520:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6657.847797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.397934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6664.400232] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6673.130682] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6680.407520] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6688.212347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6688.540392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6695.964259] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6704.384735] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6709.046741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6715.608746] Lustre: Failing over lustre-MDT0000 [ 6715.899539] Lustre: server umount lustre-MDT0000 complete [ 6716.884215] LustreError: 191517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6719.461179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6719.471965] LustreError: 191515:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6719.482435] Lustre: Skipped 2 previous similar messages [ 6719.528275] LustreError: 191515:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6724.578463] LustreError: 191521:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6724.615795] LustreError: 191521:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6733.268447] LustreError: 192521:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6733.304686] LustreError: 192521:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 6735.839183] Lustre: 190830:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789130247/real 1789130247] req@000000009bd382ca x1876038938293248/t0(0) o400->MGC192.168.201.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789130263 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6735.871502] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6737.316089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.536607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6746.629389] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6748.561628] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6751.732688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6751.760889] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6751.847320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6752.483053] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6758.024520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6765.492693] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6775.788466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6775.791903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.819911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6777.320775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6777.321553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.792840] Lustre: server umount lustre-OST0000 complete [ 6787.556784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6787.559681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6787.571318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6787.585545] Lustre: Skipped 1 previous similar message [ 6787.589653] Lustre: Skipped 1 previous similar message [ 6789.797397] Lustre: server umount lustre-MDT0000 complete [ 6792.676800] LustreError: 191520:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6792.717036] LustreError: 191520:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 16 previous similar messages [ 6794.002114] LustreError: 191503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130322 with bad export cookie 8221410329103044625 [ 6794.017555] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6794.022853] LustreError: 191503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6794.540240] Lustre: server umount lustre-MDT0001 complete [ 6809.047655] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 6812.878314] Key type lgssc unregistered [ 6813.381862] LNet: 196374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6813.398490] LNetError: 196374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6813.422430] LNet: Removed LNI 192.168.201.123@tcp [ 6814.491552] Key type .llcrypt unregistered [ 6814.495446] Key type ._llcrypt unregistered [ 6840.071349] Key type ._llcrypt registered [ 6840.076694] Key type .llcrypt registered [ 6840.197881] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 6841.030371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6841.123202] alg: No test for adler32 (adler32-zlib) [ 6842.445878] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 6842.793365] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 6844.463223] Key type lgssc registered [ 6845.944209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6854.598612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6854.646905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6856.184197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.661780] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6866.402209] LustreError: 197785:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6866.415298] LustreError: 197785:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6871.521537] LustreError: 197785:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6871.795819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6872.249644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6878.357401] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6887.273175] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6893.513604] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6901.446685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6901.997605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6909.231927] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6917.166892] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6926.601479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6931.840152] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6941.241558] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6965.371775] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6979.042075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6979.051759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6979.091698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6983.141588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6983.142886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6983.147074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6983.162512] Lustre: Skipped 1 previous similar message [ 6985.338496] Lustre: server umount lustre-OST0000 complete [ 6990.305351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6990.309932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6990.341658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6993.377763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6993.377805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6993.409369] Lustre: Skipped 1 previous similar message [ 6995.980577] Lustre: server umount lustre-MDT0000 complete [ 6998.499762] LustreError: 197781:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6998.520794] LustreError: 197781:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6999.821782] LustreError: 197764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130527 with bad export cookie 3566413575949065834 [ 6999.827867] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6999.832172] LustreError: 197764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7000.069380] Lustre: server umount lustre-MDT0001 complete [ 7012.207030] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 7015.721878] Key type lgssc unregistered [ 7016.070356] LNet: 201708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7016.082349] LNetError: 201708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7017.135463] LNet: Removed LNI 192.168.201.123@tcp [ 7018.282867] Key type .llcrypt unregistered [ 7018.287407] Key type ._llcrypt unregistered [ 7043.925197] Key type ._llcrypt registered [ 7043.929944] Key type .llcrypt registered [ 7044.099565] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 7045.236647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7045.291053] alg: No test for adler32 (adler32-zlib) [ 7046.285471] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 7046.467478] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 7048.095272] Key type lgssc registered [ 7049.525557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7059.231331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7059.247165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.765950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7065.653194] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7070.191409] LustreError: 202979:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7070.237305] LustreError: 202979:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7074.461200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7074.841656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7080.528138] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7089.649864] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7096.624830] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7104.128835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7104.588041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7105.600348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 7111.574239] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7120.625503] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7126.509425] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 7139.502659] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7149.261513] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 7155.640104] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 7170.021292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7170.022337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7170.033568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7170.056469] Lustre: Skipped 1 previous similar message [ 7170.078755] Lustre: Skipped 1 previous similar message [ 7175.142603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7175.150821] Lustre: Skipped 1 previous similar message [ 7180.261035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7180.271296] Lustre: Skipped 1 previous similar message [ 7181.279344] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7181.544872] Lustre: server umount lustre-OST0000 complete [ 7187.425302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7187.432339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7187.451488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7190.510647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7192.423953] Lustre: server umount lustre-MDT0000 complete [ 7195.616349] LustreError: 202979:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7195.639584] LustreError: 202979:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7196.856830] LustreError: 206061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130725 with bad export cookie 13578252581259202074 [ 7196.868334] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7196.872312] LustreError: 206061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7197.529768] Lustre: server umount lustre-MDT0001 complete [ 7210.472401] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 7214.097937] Key type lgssc unregistered [ 7214.442871] LNet: 207913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7214.451680] LNetError: 207913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7214.476062] LNet: Removed LNI 192.168.201.123@tcp [ 7215.544211] Key type .llcrypt unregistered [ 7215.550427] Key type ._llcrypt unregistered [ 7244.891991] Key type ._llcrypt registered [ 7244.893185] Key type .llcrypt registered [ 7244.961842] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 7246.456956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7246.597765] alg: No test for adler32 (adler32-zlib) [ 7247.721336] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 7248.065488] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 7249.663228] Key type lgssc registered [ 7251.476509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7261.551287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7261.585172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.011815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7268.504353] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7273.442895] LustreError: 209331:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7273.462982] LustreError: 209331:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7278.566561] LustreError: 209330:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7279.099303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.792098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7285.183088] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7294.672902] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7302.086593] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7310.313633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7310.718213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7318.150207] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7328.429652] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7333.256577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7354.857491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7354.875519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7354.899846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7358.957836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7358.963966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7358.978773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7360.977980] Lustre: server umount lustre-OST0000 complete [ 7366.625600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7366.635160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.651972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7366.654084] Lustre: Skipped 1 previous similar message [ 7369.192697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7369.193927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.289874] Lustre: server umount lustre-MDT0000 complete [ 7374.311570] LustreError: 210058:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.337846] LustreError: 210058:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7377.382818] LustreError: 209312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130905 with bad export cookie 17376544255821485029 [ 7377.397661] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7377.819856] Lustre: server umount lustre-MDT0001 complete [ 7393.988124] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 7399.308417] Key type lgssc unregistered [ 7399.673361] LNet: 213041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7399.682261] LNetError: 213041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7400.754190] LNet: Removed LNI 192.168.201.123@tcp [ 7401.848041] Key type .llcrypt unregistered [ 7401.849444] Key type ._llcrypt unregistered [ 7425.162980] Key type ._llcrypt registered [ 7425.166336] Key type .llcrypt registered [ 7425.301962] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 7426.550372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7426.824158] alg: No test for adler32 (adler32-zlib) [ 7427.982128] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 7428.309544] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 7430.064270] Key type lgssc registered [ 7431.938395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7441.755359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7441.784717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7443.463172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7448.914167] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7453.672507] LustreError: 214309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.694426] LustreError: 214309:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7458.786764] LustreError: 214308:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7460.222917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7460.717802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7466.464364] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7475.993340] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7483.245578] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7491.938466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7492.482362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7499.629712] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7509.702749] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7512.961030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7531.488159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7531.494841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7531.512588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7533.538950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7533.557264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7533.579047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7536.082525] Lustre: server umount lustre-OST0000 complete [ 7542.751983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7542.760377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7542.779639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7542.789365] Lustre: Skipped 1 previous similar message [ 7546.254479] Lustre: server umount lustre-MDT0000 complete [ 7548.902931] LustreError: 215986:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7548.919743] LustreError: 215986:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7550.896238] LustreError: 215043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131079 with bad export cookie 16413289220372528551 [ 7550.903444] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7550.917763] LustreError: 215043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7551.363906] Lustre: server umount lustre-MDT0001 complete [ 7564.230852] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 7567.513390] Key type lgssc unregistered [ 7567.811738] LNet: 218022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7567.821887] LNetError: 218022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7567.838045] LNet: Removed LNI 192.168.201.123@tcp [ 7568.852839] Key type .llcrypt unregistered [ 7568.856918] Key type ._llcrypt unregistered [ 7592.625170] Key type ._llcrypt registered [ 7592.627952] Key type .llcrypt registered [ 7592.741675] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 7594.258841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7594.456438] alg: No test for adler32 (adler32-zlib) [ 7595.606163] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 7595.897498] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 7597.631188] Key type lgssc registered [ 7599.068540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7608.485173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7608.517668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7610.209433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7614.984933] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7620.579404] LustreError: 219294:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7620.605608] LustreError: 219294:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7624.844532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7625.383518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7631.101709] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7641.976375] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7649.677367] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7658.098685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7658.776501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7667.884937] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7677.969734] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7680.305652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7693.939866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7694.066764] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7694.476270] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7701.106102] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7711.312795] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7713.841131] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7726.331420] Lustre: Permanently deactivating lustre-OST0001 [ 7726.356570] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7734.318822] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7741.573509] Lustre: Permanently reactivating lustre-OST0001 [ 7741.591563] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7741.595605] Lustre: Skipped 2 previous similar messages [ 7744.363898] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7744.394800] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7744.402959] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7744.420776] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7744.469430] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7754.720079] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7754.722203] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7754.730587] LustreError: Skipped 1 previous similar message [ 7754.739731] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7754.744863] Lustre: Skipped 2 previous similar messages [ 7754.760194] Lustre: Skipped 1 previous similar message [ 7760.364121] LustreError: 221117:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7760.384573] LustreError: 221117:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7760.478420] Lustre: server umount lustre-OST0001 complete [ 7765.476662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.487794] Lustre: Skipped 1 previous similar message [ 7765.496235] LustreError: 221788:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7765.496583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7765.496588] Lustre: Skipped 2 previous similar messages [ 7765.567750] LustreError: 221788:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7770.592941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7770.593417] LustreError: 221117:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7770.617073] Lustre: Skipped 2 previous similar messages [ 7770.649938] LustreError: 221117:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7775.716755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7775.726369] LustreError: 221116:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7775.736409] Lustre: Skipped 1 previous similar message [ 7775.779710] LustreError: 221116:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7779.295485] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7779.451758] Lustre: server umount lustre-OST0000 complete [ 7783.912169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7783.921151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7783.945918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7796.196819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7796.199243] Lustre: Skipped 4 previous similar messages [ 7798.240719] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7798.489049] Lustre: server umount lustre-MDT0000 complete [ 7801.323319] LustreError: 221009:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7801.367078] LustreError: 221009:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7803.442433] LustreError: 222134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131331 with bad export cookie 8942186696214898436 [ 7803.445285] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7803.459257] LustreError: 222134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7803.864499] Lustre: server umount lustre-MDT0001 complete [ 7817.541235] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 7821.127669] Key type lgssc unregistered [ 7821.462923] LNet: 224359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7821.489139] LNetError: 224359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7821.508841] LNet: Removed LNI 192.168.201.123@tcp [ 7822.557785] Key type .llcrypt unregistered [ 7822.563729] Key type ._llcrypt unregistered [ 7871.094700] Key type ._llcrypt registered [ 7871.100404] Key type .llcrypt registered [ 7871.202521] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 7872.765468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7872.922982] alg: No test for adler32 (adler32-zlib) [ 7874.160657] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 7874.564112] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 7876.240362] Key type lgssc registered [ 7877.746691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7887.388246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7887.404667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7888.631629] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7888.665897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7888.830819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7894.070867] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7903.044084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7903.203959] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7903.294942] Lustre: 226570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7903.575900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7907.846660] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7917.445395] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7926.345168] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7935.899971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7936.239129] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7936.716934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7945.445533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7946.039759] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7955.846462] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7959.331064] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7965.220390] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7973.583708] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7980.486269] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7996.877885] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7996.889071] Lustre: Skipped 1 previous similar message [ 8019.250910] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 8019.255940] Lustre: Skipped 2 previous similar messages [ 8052.027867] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 8052.036770] Lustre: Skipped 3 previous similar messages [ 8125.819280] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 8125.824535] Lustre: Skipped 8 previous similar messages [ 8135.141708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8135.160771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.163159] LustreError: Skipped 1 previous similar message [ 8135.164170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8135.188820] Lustre: Skipped 1 previous similar message [ 8140.252792] Lustre: server umount lustre-OST0000 complete [ 8145.377351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8145.397825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8145.410287] Lustre: Skipped 2 previous similar messages [ 8150.497467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8150.504196] Lustre: Skipped 1 previous similar message [ 8151.095502] Lustre: server umount lustre-MDT0000 complete [ 8155.473244] LustreError: 228442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131683 with bad export cookie 16410317348453562399 [ 8155.482906] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8155.483525] LustreError: 228442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8155.789729] Lustre: server umount lustre-MDT0001 complete [ 8167.218612] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 8170.439751] Key type lgssc unregistered [ 8170.728276] LNet: 231368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8170.733550] LNetError: 231368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8170.754248] LNet: Removed LNI 192.168.201.123@tcp [ 8171.820730] Key type .llcrypt unregistered [ 8171.822861] Key type ._llcrypt unregistered [ 8197.435863] Key type ._llcrypt registered [ 8197.438862] Key type .llcrypt registered [ 8197.582918] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 8198.869083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8199.133939] alg: No test for adler32 (adler32-zlib) [ 8200.328483] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 8200.602536] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 8202.296058] Key type lgssc registered [ 8203.229554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8210.121586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8210.138863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8211.774139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8216.087438] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8221.172884] LustreError: 232784:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8221.222394] LustreError: 232784:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8225.671131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8226.176750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8233.575517] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8243.809526] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8250.772185] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8256.491469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8256.938908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8264.196950] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8270.526966] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8281.523083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 8308.203191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8308.207796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8308.220440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8309.290623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8309.296039] Lustre: Skipped 1 previous similar message [ 8313.642431] Lustre: server umount lustre-OST0000 complete [ 8318.432618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8318.444737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8318.454704] Lustre: Skipped 1 previous similar message [ 8318.462456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8318.469287] Lustre: Skipped 1 previous similar message [ 8320.481409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8320.490163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.600988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8325.614728] Lustre: Skipped 2 previous similar messages [ 8332.255863] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8332.426726] Lustre: server umount lustre-MDT0000 complete [ 8335.847411] LustreError: 235400:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8335.895960] LustreError: 235400:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8336.904566] LustreError: 232784:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8336.938325] LustreError: 232784:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8337.160767] LustreError: 232763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131865 with bad export cookie 8961497517304400155 [ 8337.174184] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8337.182682] LustreError: 232763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8337.662873] Lustre: server umount lustre-MDT0001 complete [ 8352.967845] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 8356.309352] Key type lgssc unregistered [ 8356.759271] LNet: 236511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8356.768529] LNetError: 236511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8356.786452] LNet: Removed LNI 192.168.201.123@tcp [ 8357.757834] Key type .llcrypt unregistered [ 8357.765872] Key type ._llcrypt unregistered [ 8386.062488] Key type ._llcrypt registered [ 8386.065380] Key type .llcrypt registered [ 8386.260830] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 8388.068626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8388.159224] alg: No test for adler32 (adler32-zlib) [ 8389.342776] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 8389.765296] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 8391.512677] Key type lgssc registered [ 8393.113219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8668.070867] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 8671.941224] Key type lgssc unregistered [ 8672.296779] LNet: 238543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8672.318033] LNetError: 238543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8672.357941] LNet: Removed LNI 192.168.201.123@tcp [ 8673.529609] Key type .llcrypt unregistered [ 8673.531350] Key type ._llcrypt unregistered [ 8707.255415] Key type ._llcrypt registered [ 8707.260503] Key type .llcrypt registered [ 8707.425315] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 8730.000519] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 8731.198033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8731.274141] alg: No test for adler32 (adler32-zlib) [ 8732.273201] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 8732.513752] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 8734.207233] Key type lgssc registered [ 8735.669047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8775.651549] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8487 sec ========== 09:11:41 (1789132301) [ 8777.914917] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:11:44 (1789132304) === [ 8781.797367] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:11:48 (1789132308) === [ 8833.813504] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 8837.593453] Key type lgssc unregistered [ 8837.914336] LNet: 243792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8837.923178] LNetError: 243792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8837.946140] LNet: Removed LNI 192.168.201.123@tcp [ 8839.030140] Key type .llcrypt unregistered [ 8839.042961] Key type ._llcrypt unregistered