[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500310080 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2640MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002366] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007303] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010152] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.014065] *** VALIDATE selinux *** [ 0.022696] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027738] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030129] *** VALIDATE tmpfs *** [ 0.031453] *** VALIDATE proc *** [ 0.033095] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.041688] debug: unmapping init [mem 0xffffffffa6259000-0xffffffffa6260fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044661] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045028] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052317] rcu: Hierarchical SRCU implementation. [ 0.054580] smp: Bringing up secondary CPUs ... [ 0.055531] x86: Booting SMP configuration: [ 0.056031] .... node #0, CPUs: #1 #2 #3 [ 0.060377] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141049] node 0 deferred pages initialised in 76ms [ 0.145140] devtmpfs: initialized [ 0.147520] x86/mm: Memory block size: 128MB [ 0.152616] gcov: version magic: 0x41383552 [ 0.155431] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161313] pinctrl core: initialized pinctrl subsystem [ 0.163177] [ 0.163691] ************************************************************* [ 0.166015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169014] ** ** [ 0.171011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173014] ** ** [ 0.175014] ** This means that this kernel is built to expose internal ** [ 0.178012] ** IOMMU data structures, which may compromise security on ** [ 0.181016] ** your system. ** [ 0.183010] ** ** [ 0.185014] ** If you see this message and you are not debugging the ** [ 0.187015] ** kernel, report this immediately to your vendor! ** [ 0.190012] ** ** [ 0.191010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193013] ************************************************************* [ 0.196758] NET: Registered protocol family 16 [ 0.198455] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206035] cpuidle: using governor menu [ 0.209210] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211410] PCI: Using configuration type 1 for base access [ 0.213118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.223054] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.225023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.229027] cryptd: max_cpu_qlen set to 1000 [ 0.232275] ACPI: Added _OSI(Module Device) [ 0.233013] ACPI: Added _OSI(Processor Device) [ 0.234009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.235012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239188] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.244435] ACPI: Interpreter enabled [ 0.246067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248015] ACPI: Using IOAPIC for interrupt routing [ 0.250104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253440] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263696] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.266051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.269021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.278386] acpiphp: Slot [2] registered [ 0.279104] acpiphp: Slot [5] registered [ 0.280094] acpiphp: Slot [6] registered [ 0.282111] acpiphp: Slot [7] registered [ 0.283093] acpiphp: Slot [8] registered [ 0.284101] acpiphp: Slot [9] registered [ 0.285042] acpiphp: Slot [10] registered [ 0.286104] acpiphp: Slot [3] registered [ 0.287095] acpiphp: Slot [4] registered [ 0.288109] acpiphp: Slot [11] registered [ 0.288841] acpiphp: Slot [12] registered [ 0.289060] acpiphp: Slot [13] registered [ 0.289987] acpiphp: Slot [14] registered [ 0.291062] acpiphp: Slot [15] registered [ 0.293118] acpiphp: Slot [16] registered [ 0.294188] acpiphp: Slot [17] registered [ 0.296091] acpiphp: Slot [18] registered [ 0.296856] acpiphp: Slot [19] registered [ 0.297090] acpiphp: Slot [20] registered [ 0.298009] acpiphp: Slot [21] registered [ 0.298905] acpiphp: Slot [22] registered [ 0.300104] acpiphp: Slot [23] registered [ 0.300979] acpiphp: Slot [24] registered [ 0.301062] acpiphp: Slot [25] registered [ 0.302036] acpiphp: Slot [26] registered [ 0.302972] acpiphp: Slot [27] registered [ 0.304055] acpiphp: Slot [28] registered [ 0.304983] acpiphp: Slot [29] registered [ 0.306087] acpiphp: Slot [30] registered [ 0.307020] acpiphp: Slot [31] registered [ 0.307918] PCI host bridge to bus 0000:00 [ 0.308013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.310015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.311028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.314031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.316024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.319025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.321298] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.323942] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.327395] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.336909] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.341076] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.346018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.348016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351458] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.353835] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.356038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.357729] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.372756] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.375014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.379090] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.389018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.399017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.423020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.440444] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.452019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.476017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.484362] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.492016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.496014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.508021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.519057] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.525023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.531018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.547017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.553290] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.559014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.566019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.582018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.595427] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.601020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.606015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.618016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.625732] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.627249] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.630383] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.632263] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.634137] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.639037] iommu: Default domain type: Passthrough [ 0.641603] SCSI subsystem initialized [ 0.643137] ACPI: bus type USB registered [ 0.644075] usbcore: registered new interface driver usbfs [ 0.646079] usbcore: registered new interface driver hub [ 0.648114] usbcore: registered new device driver usb [ 0.649164] pps_core: LinuxPPS API ver. 1 registered [ 0.651009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.653049] PTP clock support registered [ 0.655064] EDAC MC: Ver: 3.0.0 [ 0.657124] PCI: Using ACPI for IRQ routing [ 0.658471] NetLabel: Initializing [ 0.659008] NetLabel: domain hash size = 128 [ 0.661013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.662075] NetLabel: unlabeled traffic allowed by default [ 0.664110] vgaarb: loaded [ 0.665195] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.667010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.674097] clocksource: Switched to clocksource kvm-clock [ 0.782514] VFS: Disk quotas dquot_6.6.0 [ 0.784137] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.786129] *** VALIDATE ramfs *** [ 0.787149] *** VALIDATE hugetlbfs *** [ 0.788568] pnp: PnP ACPI init [ 0.790556] pnp: PnP ACPI: found 6 devices [ 0.810106] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.812891] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.814234] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.815637] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.817743] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.819899] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.822521] NET: Registered protocol family 2 [ 0.824195] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.829359] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.833271] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.838568] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.842273] TCP: Hash tables configured (established 65536 bind 65536) [ 0.845244] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.848411] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.851427] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.854034] NET: Registered protocol family 1 [ 0.856411] RPC: Registered named UNIX socket transport module. [ 0.858437] RPC: Registered udp transport module. [ 0.860215] RPC: Registered tcp transport module. [ 0.861418] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.863392] NET: Registered protocol family 44 [ 0.864313] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.868026] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.870155] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.872380] PCI: CLS 0 bytes, default 64 [ 0.873650] Unpacking initramfs... [ 2.264864] debug: unmapping init [mem 0xffff8dfffcc54000-0xffff8dfffffbffff] [ 2.268765] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.270855] software IO TLB: mapped [mem 0x00000000a1000000-0x00000000a5000000] (64MB) [ 2.273630] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.808056] Initialise system trusted keyrings [ 2.809638] Key type blacklist registered [ 2.811295] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.819820] zbud: loaded [ 2.825322] *** VALIDATE nfs *** [ 2.826511] *** VALIDATE nfs4 *** [ 2.827981] pstore: using deflate compression [ 2.832165] Platform Keyring initialized [ 2.959091] NET: Registered protocol family 38 [ 2.960979] Key type asymmetric registered [ 2.962576] Asymmetric key parser 'x509' registered [ 2.964594] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.967954] io scheduler mq-deadline registered [ 2.969687] io scheduler kyber registered [ 2.971518] io scheduler bfq registered [ 2.974093] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.976809] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.979807] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.982594] ACPI: Power Button [PWRF] [ 2.988375] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.995581] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.008831] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.017526] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.049772] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.079312] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.109957] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.117709] Non-volatile memory driver v1.3 [ 3.120953] Linux agpgart interface v0.103 [ 3.162096] virtio_blk virtio1: [vda] 136552 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.165083] vda: detected capacity change from 0 to 69914624 [ 3.191126] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.194279] vdb: detected capacity change from 0 to 1073741824 [ 3.216319] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.219238] vdc: detected capacity change from 0 to 2621440000 [ 3.231559] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.233827] vdd: detected capacity change from 0 to 2621440000 [ 3.253447] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.255948] vde: detected capacity change from 0 to 4294967296 [ 3.269646] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.272394] vdf: detected capacity change from 0 to 4294967296 [ 3.282560] libphy: Fixed MDIO Bus: probed [ 3.290251] usbcore: registered new interface driver usbserial_generic [ 3.292081] usbserial: USB Serial support registered for generic [ 3.293818] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.297343] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.299140] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.301471] mousedev: PS/2 mouse device common for all mice [ 3.303982] rtc_cmos 00:05: RTC can wake from S4 [ 3.309038] rtc_cmos 00:05: registered as rtc0 [ 3.309107] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.311086] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.317489] intel_pstate: CPU model not supported [ 3.319575] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.325060] hid: raw HID events driver (C) Jiri Kosina [ 3.325177] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.327308] usbcore: registered new interface driver usbhid [ 3.331807] usbhid: USB HID core driver [ 3.332790] drop_monitor: Initializing network drop monitor service [ 3.334423] Initializing XFRM netlink socket [ 3.335754] NET: Registered protocol family 10 [ 3.338672] Segment Routing with IPv6 [ 3.340123] NET: Registered protocol family 17 [ 3.342064] mpls_gso: MPLS GSO support [ 3.347619] RAS: Correctable Errors collector initialized. [ 3.349657] AVX version of gcm_enc/dec engaged. [ 3.350839] AES CTR mode by8 optimization enabled [ 3.433381] sched_clock: Marking stable (3433353756, 0)->(4311496734, -878142978) [ 3.436724] registered taskstats version 1 [ 3.438798] Loading compiled-in X.509 certificates [ 3.440658] zswap: loaded using pool lzo/zbud [ 3.469853] Key type big_key registered [ 3.483176] Key type encrypted registered [ 3.484827] ima: No TPM chip found, activating TPM-bypass! [ 3.486812] ima: Allocated hash algorithm: sha1 [ 3.488598] ima: No architecture policies found [ 3.490564] evm: Initialising EVM extended attributes: [ 3.492519] evm: security.selinux [ 3.493794] evm: security.ima [ 3.494868] evm: security.capability [ 3.496248] evm: HMAC attrs: 0x1 [ 3.498695] rtc_cmos 00:05: setting system clock to 2026-06-11 22:55:17 UTC (1781218517) [ 3.505077] debug: unmapping init [mem 0xffffffffa7203000-0xffffffffa73fffff] [ 3.508235] debug: unmapping init [mem 0xffffffffa5f82000-0xffffffffa6258fff] [ 3.517095] Write protecting the kernel read-only data: 28672k [ 3.520382] debug: unmapping init [mem 0xffffffffa4603000-0xffffffffa47fffff] [ 3.523282] debug: unmapping init [mem 0xffffffffa4f14000-0xffffffffa4ffffff] [ 3.559788] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.568674] systemd[1]: Detected virtualization kvm. [ 3.570943] systemd[1]: Detected architecture x86-64. [ 3.572851] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.600643] systemd[1]: No hostname configured. [ 3.602192] systemd[1]: Set hostname to . [ 3.603936] random: systemd: uninitialized urandom read (16 bytes read) [ 3.608616] systemd[1]: Initializing machine ID from random generator. [ 3.664878] random: ln: uninitialized urandom read (6 bytes read) [ 3.742545] random: systemd: uninitialized urandom read (16 bytes read) [ 3.745933] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.752853] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.759827] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.459400] device-mapper: uevent: version 1.0.3 [ 4.461514] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.277318] virtio_net virtio0 ens2: renamed from eth0 [ 5.292377] random: fast init done [ 5.299393] scsi host0: ata_piix [ 5.331901] scsi host1: ata_piix [ 5.333868] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.336378] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.156781] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.004425] random: crng init done [ 10.006320] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.547683] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.696838] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.990177] SELinux: Disabled at runtime. [ 12.048959] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.056385] systemd[1]: Detected virtualization kvm. [ 12.057667] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.506455] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.509511] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.513446] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.517188] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.519261] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.529625] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.535438] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables...[ 12.628613] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.963177] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.235025] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.304726] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.381799] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.392335] EDAC sbridge: Ver: 1.1.2 [ 14.992871] Key type dns_resolver registered [ 15.285629] NFS: Registering the id_resolver key type [ 15.287655] Key type id_resolver registered [ 15.289156] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg122-server login: [ 43.128343] libcfs: loading out-of-tree module taints kernel. [ 43.277681] Key type ._llcrypt registered [ 43.279650] Key type .llcrypt registered [ 43.428237] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 63.189789] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 65.410781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 65.441720] alg: No test for adler32 (adler32-zlib) [ 66.792850] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 68.033499] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 69.911240] Key type lgssc registered [ 72.224228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.650902] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 107.617387] hrtimer: interrupt took 7406211 ns [ 142.453661] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 158.031229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 158.073555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.700390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 159.789862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 159.957529] Lustre: lustre-MDT0000: new disk, initializing [ 160.119112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.149230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.098667] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.214837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.328338] Lustre: 6514:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 181.394135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 181.398599] Lustre: Skipped 1 previous similar message [ 181.628176] Lustre: lustre-MDT0001: new disk, initializing [ 181.752974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 181.794383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 181.830535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 187.929420] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.422446] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.971958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.317100] Lustre: lustre-OST0000: new disk, initializing [ 205.331757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 205.349077] Lustre: 8422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.455236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.903282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.910538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.972183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 212.273850] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.380768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.595611] Lustre: lustre-OST0001: new disk, initializing [ 228.602682] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 228.618158] Lustre: 9478:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.745285] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.591605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 235.607091] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 235.694311] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.696568] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 249.969358] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 261.916532] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 270.558416] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing check_logdir /tmp/testlogs/ [ 277.047555] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing yml_node [ 282.101374] Lustre: DEBUG MARKER: Client: 2.17.53.74 [ 285.215741] Lustre: DEBUG MARKER: MDS: 2.17.53.74 [ 288.172285] Lustre: DEBUG MARKER: OSS: 2.17.53.74 [ 289.925997] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 19:00:03 EDT 2026 [ 312.126718] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 313.670327] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.208442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 330.222652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.239738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.769810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.769899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.814224] Lustre: Skipped 3 previous similar messages [ 336.580505] Lustre: server umount lustre-MDT0000 complete [ 341.126653] LustreError: 8421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218855 with bad export cookie 10958971576108349033 [ 341.133832] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 341.139995] LustreError: 8421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 343.009213] LustreError: 8430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.010669] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.022467] LustreError: 8430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 343.059675] Lustre: Skipped 2 previous similar messages [ 343.072653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.083132] Lustre: Skipped 1 previous similar message [ 346.017887] LustreError: 10039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.017977] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.040315] LustreError: 10039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 346.069750] Lustre: Skipped 1 previous similar message [ 347.761442] Lustre: server umount lustre-MDT0001 complete [ 352.561298] Lustre: server umount lustre-OST0000 complete [ 357.421902] Lustre: server umount lustre-OST0001 complete [ 365.423286] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 375.125727] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 417.945996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.352680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 418.409044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 418.571548] Lustre: lustre-MDT0000: new disk, initializing [ 418.704396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.729435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 425.169143] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 438.304242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.536787] Lustre: 15013:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 438.643421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 438.647497] Lustre: Skipped 1 previous similar message [ 438.837068] Lustre: lustre-MDT0001: new disk, initializing [ 439.010902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 439.049622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 439.067508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 444.671905] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 452.808128] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 457.278427] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 464.986197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.360658] Lustre: lustre-OST0000: new disk, initializing [ 465.366947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.384389] Lustre: 16755:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.512575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.305316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 467.313510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.463792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 474.031602] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 486.287762] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 493.809677] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 494.179688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 501.263565] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 501.679420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 508.394365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.395517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 508.403650] Lustre: Skipped 1 previous similar message [ 508.404427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 518.628348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 518.634316] Lustre: Skipped 3 previous similar messages [ 519.136056] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 519.522857] Lustre: server umount lustre-OST0000 complete [ 526.309982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 526.316738] LustreError: Skipped 1 previous similar message [ 526.322719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.351333] Lustre: Skipped 1 previous similar message [ 531.139484] Lustre: server umount lustre-MDT0000 complete [ 533.996270] LustreError: 15805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.015750] LustreError: 15805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 535.036733] LustreError: 15003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219049 with bad export cookie 10958971576108350559 [ 535.049769] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.050299] LustreError: 15003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 535.425757] Lustre: server umount lustre-MDT0001 complete [ 539.460308] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 19:04:12 (1781219052) [ 546.886901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.280397] LustreError: 18877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.348667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.831268] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 552.420852] LustreError: 18878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.861199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.972674] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 572.201109] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 576.903369] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 584.647461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 591.937764] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 598.554859] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 607.714217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 607.725963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.730141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 607.730152] Lustre: Skipped 4 previous similar messages [ 607.761318] Lustre: Skipped 2 previous similar messages [ 613.360880] Lustre: server umount lustre-OST0000 complete [ 622.564729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 623.984340] Lustre: server umount lustre-MDT0000 complete [ 628.194252] LustreError: 19861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.218360] LustreError: 19861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 628.337532] LustreError: 18857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219142 with bad export cookie 10958971576108351448 [ 628.347371] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 628.357518] LustreError: 18857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 629.040250] Lustre: server umount lustre-MDT0001 complete [ 641.941719] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 645.563565] Key type lgssc unregistered [ 646.125441] LNet: 22054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.132486] LNetError: 22054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 646.144899] LNet: Removed LNI 192.168.201.122@tcp [ 647.734290] Key type .llcrypt unregistered [ 647.737393] Key type ._llcrypt unregistered [ 676.240624] Key type ._llcrypt registered [ 676.248941] Key type .llcrypt registered [ 676.416364] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 679.053516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 679.094866] alg: No test for adler32 (adler32-zlib) [ 680.454688] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 680.968054] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 682.799203] Key type lgssc registered [ 684.248804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 693.381679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 693.421228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.114284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 700.344773] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 705.512049] LustreError: 23456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 705.531226] LustreError: 23456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 710.206242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.523678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 715.578380] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 722.108488] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 726.666323] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 733.676771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 734.054656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 740.285890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 741.135598] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 747.961915] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 771.055595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 771.062539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 771.064860] Lustre: Skipped 1 previous similar message [ 771.065409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 771.095146] Lustre: Skipped 1 previous similar message [ 772.636197] Lustre: server umount lustre-OST0000 complete [ 777.190312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 777.198659] LustreError: Skipped 1 previous similar message [ 777.205077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 777.216953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 781.289939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.292887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 781.315124] Lustre: Skipped 1 previous similar message [ 783.189840] Lustre: server umount lustre-MDT0000 complete [ 786.400587] LustreError: 23456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 786.421803] LustreError: 23456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 788.139331] LustreError: 23436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219302 with bad export cookie 13952436492806595807 [ 788.145536] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 788.149907] LustreError: 23436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 788.517418] Lustre: server umount lustre-MDT0001 complete [ 802.227746] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 805.533849] Key type lgssc unregistered [ 805.924711] LNet: 27076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.929668] LNetError: 27076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.948342] LNet: Removed LNI 192.168.201.122@tcp [ 807.140868] Key type .llcrypt unregistered [ 807.145824] Key type ._llcrypt unregistered [ 835.105471] Key type ._llcrypt registered [ 835.109305] Key type .llcrypt registered [ 835.234702] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 836.996530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 837.038434] alg: No test for adler32 (adler32-zlib) [ 838.057821] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 838.310677] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 840.024093] Key type lgssc registered [ 841.465438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 850.941640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 850.953977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.685630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.172869] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 862.179697] LustreError: 28471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 862.209302] LustreError: 28471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 867.307648] LustreError: 28472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 871.182388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.982365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 876.506688] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 882.656055] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 887.286880] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 903.372151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 903.706146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.267319] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 916.481181] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 923.200210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 931.297441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 931.307252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.332186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.354376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.358712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.378823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 933.384871] Lustre: Skipped 1 previous similar message [ 934.639257] Lustre: server umount lustre-OST0000 complete [ 943.584708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 943.590214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.598530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.602163] Lustre: Skipped 1 previous similar message [ 945.659252] Lustre: server umount lustre-MDT0000 complete [ 948.705344] LustreError: 28472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.745995] LustreError: 28472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 950.119960] LustreError: 28452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219464 with bad export cookie 15055823146772535195 [ 950.150913] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 950.540807] Lustre: server umount lustre-MDT0001 complete [ 963.238881] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 965.816134] Key type lgssc unregistered [ 966.141818] LNet: 31995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.147787] LNetError: 31995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.163493] LNet: Removed LNI 192.168.201.122@tcp [ 967.034929] Key type .llcrypt unregistered [ 967.036805] Key type ._llcrypt unregistered [ 997.255363] Key type ._llcrypt registered [ 997.258339] Key type .llcrypt registered [ 997.364252] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 998.585574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.684065] alg: No test for adler32 (adler32-zlib) [ 999.845144] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1000.029758] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1001.663262] Key type lgssc registered [ 1002.949795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1012.042886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1012.062594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.708965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.051820] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1023.994104] LustreError: 33404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.025452] LustreError: 33404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1028.234351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.651055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1032.915913] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1039.257183] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1044.341055] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1052.391347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1052.812972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1058.052328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1059.637521] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1066.178168] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1078.759888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1078.777447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.785561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1078.852863] Lustre: Skipped 1 previous similar message [ 1082.280962] Lustre: server umount lustre-OST0000 complete [ 1088.999068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.004400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1089.017983] Lustre: Skipped 1 previous similar message [ 1089.029330] Lustre: Skipped 1 previous similar message [ 1094.053567] Lustre: server umount lustre-MDT0000 complete [ 1094.132251] LustreError: 35558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.151122] LustreError: 35558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1098.208131] LustreError: 33405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.366577] LustreError: 33384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219612 with bad export cookie 2174714187484640439 [ 1098.377404] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1098.379396] LustreError: 33384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1098.837236] Lustre: server umount lustre-MDT0001 complete [ 1112.814957] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 1117.165733] Key type lgssc unregistered [ 1117.556275] LNet: 36583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.565931] LNetError: 36583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1117.598892] LNet: Removed LNI 192.168.201.122@tcp [ 1119.006865] Key type .llcrypt unregistered [ 1119.011712] Key type ._llcrypt unregistered [ 1148.343350] Key type ._llcrypt registered [ 1148.347112] Key type .llcrypt registered [ 1148.529699] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 1149.784405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1149.796184] alg: No test for adler32 (adler32-zlib) [ 1150.928410] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1151.243547] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1152.863189] Key type lgssc registered [ 1153.873022] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1163.186264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1163.236710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.682343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1168.919892] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1174.062909] LustreError: 37980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.076723] LustreError: 37980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1177.945128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.320970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1182.991321] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1189.469146] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1193.577297] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1199.889466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1200.290838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.992303] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1209.527247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1216.043288] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1224.165679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1224.169333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.181766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1224.778718] Lustre: lustre-OST0000: Not available for connect from 192.168.201.22@tcp (stopping) [ 1225.191888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1225.205870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.222697] Lustre: server umount lustre-OST0000 complete [ 1250.275766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1250.284710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.319968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1250.328989] Lustre: Skipped 2 previous similar messages [ 1255.979933] Lustre: server umount lustre-MDT0000 complete [ 1256.928677] LustreError: 37980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.945987] LustreError: 37980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1260.109589] LustreError: 38696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219774 with bad export cookie 1292688857894954923 [ 1260.116586] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1260.120627] LustreError: 38696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1266.992705] Lustre: server umount lustre-MDT0001 complete [ 1279.534531] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 1283.195977] Key type lgssc unregistered [ 1283.481944] LNet: 41331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.486346] LNetError: 41331:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.505279] LNet: Removed LNI 192.168.201.122@tcp [ 1284.677181] Key type .llcrypt unregistered [ 1284.682316] Key type ._llcrypt unregistered [ 1310.340919] Key type ._llcrypt registered [ 1310.343354] Key type .llcrypt registered [ 1310.498723] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 1311.597368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1311.640192] alg: No test for adler32 (adler32-zlib) [ 1312.818249] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1313.056275] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1314.735214] Key type lgssc registered [ 1316.021150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1323.743751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1323.765946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.477143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1330.685729] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1335.790985] LustreError: 42730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.837985] LustreError: 42730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1340.910763] LustreError: 42729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.429972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.952256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1347.233100] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1355.460348] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1360.133640] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1367.953449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1368.364162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1370.381888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1376.095603] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1382.961494] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1396.274652] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.22@tcp (stopping) [ 1397.220685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.227266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1397.243720] Lustre: Skipped 1 previous similar message [ 1400.872921] Lustre: server umount lustre-MDT0000 complete [ 1401.363072] LustreError: 44881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.085454] LustreError: 42709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219919 with bad export cookie 12341291667482709561 [ 1405.109463] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1405.119215] LustreError: 42709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1406.477484] LustreError: 43657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1406.502519] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.22@tcp (stopping) [ 1406.518552] LustreError: 43657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1406.538064] Lustre: Skipped 2 previous similar messages [ 1407.487083] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.505031] Lustre: Skipped 1 previous similar message [ 1410.465508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1410.476778] Lustre: Skipped 1 previous similar message [ 1411.582972] Lustre: server umount lustre-MDT0001 complete [ 1429.200929] Lustre: server umount lustre-OST0000 complete [ 1446.736714] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 1449.654723] Key type lgssc unregistered [ 1450.043794] LNet: 46239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.057962] LNetError: 46239:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.110659] LNet: Removed LNI 192.168.201.122@tcp [ 1452.277602] Key type .llcrypt unregistered [ 1452.280056] Key type ._llcrypt unregistered [ 1483.801899] Key type ._llcrypt registered [ 1483.802958] Key type .llcrypt registered [ 1484.038416] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 1485.691037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1485.751480] alg: No test for adler32 (adler32-zlib) [ 1486.861445] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1487.249689] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1489.193148] Key type lgssc registered [ 1490.729301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1499.773713] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1499.813624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1515.617302] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1520.735320] LustreError: 47610:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 1547.488228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1554.225937] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1560.202512] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1724.621045] Lustre: server umount lustre-OST0000 complete [ 1742.034714] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 1745.536515] Key type lgssc unregistered [ 1745.871708] LNet: 49157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.885992] LNetError: 49157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.907272] LNet: Removed LNI 192.168.201.122@tcp [ 1746.888776] Key type .llcrypt unregistered [ 1746.898348] Key type ._llcrypt unregistered [ 1772.572533] Key type ._llcrypt registered [ 1772.575164] Key type .llcrypt registered [ 1772.684906] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 1773.604024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1773.798442] alg: No test for adler32 (adler32-zlib) [ 1774.821222] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1775.028516] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1776.735347] Key type lgssc registered [ 1778.915776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1789.261129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1789.280971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.685753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.740744] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1800.167130] LustreError: 50556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.192877] LustreError: 50556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1805.294202] LustreError: 50557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.018240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.319952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1812.832481] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1821.420741] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1826.579984] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1837.532758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1837.831485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.670448] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1854.224834] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1857.517221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.539842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1857.552372] LustreError: 52156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1857.837624] Lustre: server umount lustre-OST0000 complete [ 1863.660874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1863.679366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.711986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1865.696541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.699281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1865.716130] Lustre: Skipped 1 previous similar message [ 1869.065817] Lustre: server umount lustre-MDT0000 complete [ 1870.819168] LustreError: 51267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1870.844396] LustreError: 51267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1873.201385] LustreError: 50537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220387 with bad export cookie 14808485736063407163 [ 1873.213275] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1873.220264] LustreError: 50537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1873.708633] Lustre: server umount lustre-MDT0001 complete [ 1887.976504] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 1891.196876] Key type lgssc unregistered [ 1891.448373] LNet: 53735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1891.452353] LNetError: 53735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1891.465259] LNet: Removed LNI 192.168.201.122@tcp [ 1892.115070] Key type .llcrypt unregistered [ 1892.118700] Key type ._llcrypt unregistered [ 1917.389939] Key type ._llcrypt registered [ 1917.392452] Key type .llcrypt registered [ 1917.541708] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 1918.751484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1918.938042] alg: No test for adler32 (adler32-zlib) [ 1920.045514] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1920.278296] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1921.943476] Key type lgssc registered [ 1922.962848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1929.845731] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1929.854401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1945.440573] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1950.559222] LustreError: 55104:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 1977.247260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.486770] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1991.261956] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1999.401888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.081851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2000.250302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2005.482440] LustreError: 55130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.523685] LustreError: 55130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2005.748259] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2010.596252] LustreError: 55129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.624034] LustreError: 55130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.657362] LustreError: 55130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2016.641263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.737182] LustreError: 56029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.759102] LustreError: 56029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2017.009561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2022.631550] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2028.722318] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2033.071923] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2036.193702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2036.203780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.231866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2037.730500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.758608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2042.854431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2042.864429] Lustre: Skipped 1 previous similar message [ 2047.976045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2047.990981] Lustre: Skipped 1 previous similar message [ 2050.015506] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2050.161878] Lustre: server umount lustre-OST0000 complete [ 2073.575126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2073.578615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.589402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2073.608051] Lustre: Skipped 1 previous similar message [ 2073.625509] Lustre: Skipped 1 previous similar message [ 2079.713085] LustreError: 56741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.740202] LustreError: 56741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2081.480501] Lustre: server umount lustre-MDT0000 complete [ 2085.833142] LustreError: 55111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220599 with bad export cookie 14579319511477369886 [ 2085.833790] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2085.841701] LustreError: 55111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2100.191148] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2100.873617] Lustre: server umount lustre-MDT0001 complete [ 2116.567055] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2120.566792] Key type lgssc unregistered [ 2121.167289] LNet: 58497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.177480] LNetError: 58497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.207935] LNet: Removed LNI 192.168.201.122@tcp [ 2122.842242] Key type .llcrypt unregistered [ 2122.843890] Key type ._llcrypt unregistered [ 2148.551344] Key type ._llcrypt registered [ 2148.552682] Key type .llcrypt registered [ 2148.672397] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2149.632692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2149.862396] alg: No test for adler32 (adler32-zlib) [ 2151.062179] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2151.250440] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2152.848323] Key type lgssc registered [ 2154.380526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2162.377813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2162.391916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.943642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2168.563750] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2173.425246] LustreError: 59894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2173.462585] LustreError: 59894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2177.007314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.286225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2182.310646] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2189.161931] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2193.617151] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2201.833616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2202.035785] LustreError: 61489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.274094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2208.487255] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2208.758202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2214.962470] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2234.338277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2234.342527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.346803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2234.374080] Lustre: Skipped 1 previous similar message [ 2239.459726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2239.480552] Lustre: Skipped 2 previous similar messages [ 2239.901526] Lustre: server umount lustre-OST0000 complete [ 2244.068715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2244.073720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.099200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.699436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.714465] Lustre: Skipped 2 previous similar messages [ 2254.817174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2254.828737] Lustre: Skipped 1 previous similar message [ 2258.400943] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2258.780705] Lustre: server umount lustre-MDT0000 complete [ 2259.936568] LustreError: 59895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.974289] LustreError: 59895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2264.038520] LustreError: 59890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.066711] LustreError: 59890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2265.049964] LustreError: 59876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220779 with bad export cookie 9126678422255734962 [ 2265.055124] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2265.058985] LustreError: 59876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2265.350908] Lustre: server umount lustre-MDT0001 complete [ 2278.656686] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2281.904924] Key type lgssc unregistered [ 2282.335140] LNet: 63092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.351569] LNetError: 63092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2282.380382] LNet: Removed LNI 192.168.201.122@tcp [ 2283.296725] Key type .llcrypt unregistered [ 2283.298317] Key type ._llcrypt unregistered [ 2312.525334] Key type ._llcrypt registered [ 2312.527016] Key type .llcrypt registered [ 2312.695270] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2314.328632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2314.448306] alg: No test for adler32 (adler32-zlib) [ 2315.599824] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2315.992071] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2317.855301] Key type lgssc registered [ 2320.414770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2332.030287] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 19:34:04 (1781220844) [ 2341.267743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2341.281868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2342.748066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.648133] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2352.099494] LustreError: 64785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.108806] LustreError: 64785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2356.500183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2356.879363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2362.559512] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2368.769278] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2372.911716] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2379.098476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2379.583850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.053821] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2387.798330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2392.872935] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2399.379476] Lustre: Failing over lustre-MDT0000 [ 2399.710818] Lustre: server umount lustre-MDT0000 complete [ 2400.809279] LustreError: 66510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.784151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2402.794451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.805810] LustreError: 65243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.311185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.901160] LustreError: 64780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.915280] LustreError: 64780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2409.221970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2409.438419] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.605922] Lustre: *** cfs_fail_loc=135, val=0*** [ 2409.675707] LustreError: 67532:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2409.679047] LustreError: 67532:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.122@tcp: cfg command failed: rc = -2 [ 2409.683140] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2409.683140] [ 2409.687118] LustreError: MGC192.168.201.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2409.693079] LustreError: 67521:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2409.697473] LustreError: 67521:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2409.703607] LustreError: 67521:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2409.707842] LustreError: 67521:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2409.727916] Lustre: server umount lustre-MDT0000 complete [ 2409.729833] LustreError: 67521:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2411.019212] LustreError: 66510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.050323] LustreError: 66510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2416.823129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.058720] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2417.071786] LustreError: 64778:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2417.318377] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.22@tcp (not set up) [ 2417.451213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.532361] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2421.325560] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2422.652465] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2422.792847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2422.845253] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2422.939964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2432.996993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2433.009583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2433.023993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2433.049262] Lustre: Skipped 2 previous similar messages [ 2437.155287] Lustre: server umount lustre-OST0000 complete [ 2443.237047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2443.237195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2443.242871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2443.242880] Lustre: Skipped 1 previous similar message [ 2443.267077] Lustre: Skipped 1 previous similar message [ 2447.388619] Lustre: server umount lustre-MDT0000 complete [ 2448.358788] LustreError: 65243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.384414] LustreError: 65243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2451.351855] LustreError: 64766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220965 with bad export cookie 11530619187623209254 [ 2451.364477] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2451.630489] Lustre: server umount lustre-MDT0001 complete [ 2463.230324] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2466.637473] Key type lgssc unregistered [ 2467.038909] LNet: 69276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.046469] LNetError: 69276:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.092200] LNet: Removed LNI 192.168.201.122@tcp [ 2468.172493] Key type .llcrypt unregistered [ 2468.174738] Key type ._llcrypt unregistered [ 2495.738333] Key type ._llcrypt registered [ 2495.743256] Key type .llcrypt registered [ 2495.935331] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2497.269787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2497.468303] alg: No test for adler32 (adler32-zlib) [ 2498.633485] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2499.089278] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2500.984353] Key type lgssc registered [ 2502.822644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2510.562787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2510.577171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.344536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.147289] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2522.597150] LustreError: 70672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.623601] LustreError: 70672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2526.215479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.651499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2531.282608] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2537.666440] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2542.553685] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2550.421344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2550.934758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.065271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2558.534272] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2566.400995] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2574.605984] Lustre: Failing over lustre-MDT0000 [ 2575.233969] Lustre: server umount lustre-MDT0000 complete [ 2577.891285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2577.900326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.908992] LustreError: 70667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.919139] Lustre: Skipped 2 previous similar messages [ 2577.932240] LustreError: 70667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2579.502739] LustreError: 70668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.540848] LustreError: 70668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2583.024246] LustreError: 71887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.657671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2585.742786] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2585.863582] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2586.018561] LustreError: 73422:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2586.028271] LustreError: 73422:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.122@tcp: cfg command failed: rc = -9 [ 2586.040355] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2586.040355] [ 2586.052081] LustreError: MGC192.168.201.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2586.065182] LustreError: 73410:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2586.070722] LustreError: 73410:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 2586.079493] LustreError: 73410:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2586.090576] LustreError: 73410:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2586.120396] Lustre: server umount lustre-MDT0000 complete [ 2586.122918] LustreError: 73410:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2588.131519] LustreError: 70673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.145392] LustreError: 70673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2593.026946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.141400] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2593.153659] LustreError: 70664:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2593.263180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2593.282843] Lustre: Skipped 2 previous similar messages [ 2593.536549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.600577] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2594.837401] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2598.754681] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2598.900756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2598.976483] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2599.073595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2608.611332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2608.620772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.652796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2613.577626] Lustre: server umount lustre-OST0000 complete [ 2619.363491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2619.365673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.380288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2619.387990] Lustre: Skipped 2 previous similar messages [ 2619.418335] Lustre: Skipped 3 previous similar messages [ 2624.281660] Lustre: server umount lustre-MDT0000 complete [ 2624.495629] LustreError: 70673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.511951] LustreError: 70673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2628.250402] LustreError: 70653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221142 with bad export cookie 229020522468493115 [ 2628.269204] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2628.560798] Lustre: server umount lustre-MDT0001 complete [ 2640.780934] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2645.047523] Key type lgssc unregistered [ 2645.450229] LNet: 75170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2645.456695] LNetError: 75170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2645.473480] LNet: Removed LNI 192.168.201.122@tcp [ 2646.719043] Key type .llcrypt unregistered [ 2646.724126] Key type ._llcrypt unregistered [ 2663.943717] Key type ._llcrypt registered [ 2663.948278] Key type .llcrypt registered [ 2664.111508] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2664.560417] Key type .llcrypt unregistered [ 2664.561911] Key type ._llcrypt unregistered [ 2690.566578] Key type ._llcrypt registered [ 2690.568911] Key type .llcrypt registered [ 2690.683706] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2691.936460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2692.118813] alg: No test for adler32 (adler32-zlib) [ 2693.284626] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2693.536546] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2695.279307] Key type lgssc registered [ 2696.815808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2704.607772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2704.649857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.171118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.427574] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2716.651762] LustreError: 77051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.672576] LustreError: 77051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2721.540132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.761408] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2722.081685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2727.184530] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2733.276702] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2737.295745] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2744.521386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2744.998175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2751.043550] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2751.169243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2756.848776] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2771.942696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2771.947552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.949620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.959606] Lustre: Skipped 1 previous similar message [ 2773.725407] Lustre: server umount lustre-OST0000 complete [ 2778.595211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2778.599822] LustreError: Skipped 1 previous similar message [ 2778.603394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.625772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2778.636565] Lustre: Skipped 1 previous similar message [ 2782.178666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.179443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.029765] Lustre: server umount lustre-MDT0000 complete [ 2787.307166] LustreError: 77051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.344238] LustreError: 77051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2788.129530] LustreError: 77031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221302 with bad export cookie 4372234346070846280 [ 2788.148481] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2788.149939] LustreError: 77031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2788.473987] Lustre: server umount lustre-MDT0001 complete [ 2801.313844] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2804.922676] Key type lgssc unregistered [ 2805.204797] LNet: 80224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2805.211950] LNetError: 80224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2805.232209] LNet: Removed LNI 192.168.201.122@tcp [ 2806.092929] Key type .llcrypt unregistered [ 2806.102446] Key type ._llcrypt unregistered [ 2830.404651] Key type ._llcrypt registered [ 2830.406164] Key type .llcrypt registered [ 2830.537554] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2831.533259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2831.841105] alg: No test for adler32 (adler32-zlib) [ 2832.910542] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2833.118519] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2834.815200] Key type lgssc registered [ 2836.047540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2845.168495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2845.195163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.897883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.317717] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2856.417372] LustreError: 81620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.447758] LustreError: 81620:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2861.540264] LustreError: 81621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.144791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.512618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2869.011097] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2875.109628] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2879.568484] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2887.343634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2887.896506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.043335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2894.930195] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2901.693395] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2909.671203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2909.683858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.704928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2910.239933] Lustre: server umount lustre-OST0000 complete [ 2915.810149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.818194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2915.823035] Lustre: Skipped 1 previous similar message [ 2915.837121] Lustre: Skipped 1 previous similar message [ 2920.929089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2920.937353] Lustre: Skipped 1 previous similar message [ 2925.034250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2925.058026] Lustre: Skipped 1 previous similar message [ 2925.184655] Lustre: server umount lustre-MDT0000 complete [ 2929.717834] LustreError: 81601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221443 with bad export cookie 9545994427760998718 [ 2929.753317] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2930.392823] Lustre: server umount lustre-MDT0001 complete [ 2944.946087] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2948.254511] Key type lgssc unregistered [ 2948.618531] LNet: 84797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2948.624152] LNetError: 84797:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2948.637523] LNet: Removed LNI 192.168.201.122@tcp [ 2949.593977] Key type .llcrypt unregistered [ 2949.604366] Key type ._llcrypt unregistered [ 2975.398818] Key type ._llcrypt registered [ 2975.402060] Key type .llcrypt registered [ 2975.679854] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2977.211471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2977.304218] alg: No test for adler32 (adler32-zlib) [ 2978.447290] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2978.831300] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2980.635679] Key type lgssc registered [ 2982.143377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2991.226578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2991.255677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.842521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.105178] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3002.340145] LustreError: 86199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.353669] LustreError: 86199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3006.144160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3006.606289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3012.271392] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3019.804189] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3023.620974] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3031.066583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3031.371983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3037.591355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3037.852688] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3044.511832] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3063.280515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3063.282059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.295439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3063.309139] Lustre: Skipped 1 previous similar message [ 3063.331027] Lustre: Skipped 1 previous similar message [ 3064.502424] Lustre: server umount lustre-OST0000 complete [ 3073.003773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3073.009438] LustreError: Skipped 1 previous similar message [ 3073.015718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3073.049480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3076.589488] Lustre: server umount lustre-MDT0000 complete [ 3078.631263] LustreError: 86909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.669777] LustreError: 86909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3083.138642] LustreError: 86178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221597 with bad export cookie 4002201967261701237 [ 3083.151145] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3083.607727] Lustre: server umount lustre-MDT0001 complete [ 3099.646092] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 3102.555906] Key type lgssc unregistered [ 3102.872257] LNet: 89397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3102.902945] LNetError: 89397:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3102.922958] LNet: Removed LNI 192.168.201.122@tcp [ 3103.651228] Key type .llcrypt unregistered [ 3103.654510] Key type ._llcrypt unregistered [ 3134.097502] Key type ._llcrypt registered [ 3134.102799] Key type .llcrypt registered [ 3134.210475] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3135.489464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3135.665394] alg: No test for adler32 (adler32-zlib) [ 3136.695238] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 3136.995679] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 3138.743328] Key type lgssc registered [ 3140.033273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3149.652949] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3149.690988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3165.344292] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3170.527247] LustreError: 90768:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 3197.279211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3203.138751] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3208.702249] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3216.249173] Lustre: server umount lustre-OST0000 complete [ 3226.435496] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 19:48:59 (1781221739) [ 3234.930475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.404758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.419110] LustreError: 92296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.453213] LustreError: 92296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3241.735268] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3245.555269] LustreError: 92295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.656882] LustreError: 92296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.640306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.186330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3258.621450] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3265.798954] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3270.941965] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3279.060452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3279.733760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.701805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3287.482825] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3294.079324] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3311.584403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3311.597164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3311.610353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3313.632466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3313.643072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3313.643395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.666871] Lustre: Skipped 1 previous similar message [ 3316.872629] Lustre: server umount lustre-OST0000 complete [ 3323.874667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3323.901670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3323.907594] Lustre: Skipped 1 previous similar message [ 3327.138668] Lustre: server umount lustre-MDT0000 complete [ 3328.993387] LustreError: 92295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.014355] LustreError: 92295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3331.672356] LustreError: 92275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221845 with bad export cookie 12732870978085104498 [ 3331.680632] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3331.687960] LustreError: 92275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3332.490912] Lustre: server umount lustre-MDT0001 complete [ 3344.969842] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 3348.210032] Key type lgssc unregistered [ 3348.512656] LNet: 95633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3348.520512] LNetError: 95633:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3348.538777] LNet: Removed LNI 192.168.201.122@tcp [ 3349.658194] Key type .llcrypt unregistered [ 3349.660489] Key type ._llcrypt unregistered [ 3395.010060] Key type ._llcrypt registered [ 3395.014280] Key type .llcrypt registered [ 3395.191664] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3396.443000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3396.606657] alg: No test for adler32 (adler32-zlib) [ 3397.729987] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 3397.936413] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 3399.583216] Key type lgssc registered [ 3401.255158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3411.039490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3411.059862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.719827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3417.502706] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3422.203891] LustreError: 97805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.238402] LustreError: 97805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3425.932733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.262167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3430.895230] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3437.544462] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3443.365495] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3452.443325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3452.887847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3457.095835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3460.730035] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3466.448268] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3480.035973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3480.045267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.076103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3481.569094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3481.579529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3481.579590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3484.047257] Lustre: server umount lustre-OST0000 complete [ 3491.820137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.841362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3491.855720] Lustre: Skipped 2 previous similar messages [ 3495.010502] Lustre: server umount lustre-MDT0000 complete [ 3496.931395] LustreError: 99958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3496.962229] LustreError: 99958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3499.507166] LustreError: 97785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222013 with bad export cookie 17138290437098003943 [ 3499.514623] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3500.242340] Lustre: server umount lustre-MDT0001 complete [ 3512.564587] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 3515.626218] Key type lgssc unregistered [ 3515.946800] LNet: 100985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3515.951719] LNetError: 100985:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3515.961855] LNet: Removed LNI 192.168.201.122@tcp [ 3516.905865] Key type .llcrypt unregistered [ 3516.907719] Key type ._llcrypt unregistered [ 3536.089542] Key type ._llcrypt registered [ 3536.091566] Key type .llcrypt registered [ 3536.254340] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3537.700066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3537.723422] alg: No test for adler32 (adler32-zlib) [ 3539.265546] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 3539.506942] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 3541.263454] Key type lgssc registered [ 3542.735038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3552.559725] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3552.613890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3568.287575] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3573.411326] LustreError: 102225:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 3600.223224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3610.159230] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3617.403379] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3626.890378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.328461] LustreError: MGC192.168.201.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3627.341315] LustreError: 103137:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3627.350438] LustreError: 103137:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3627.376735] LustreError: 103137:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3627.499633] Lustre: server umount lustre-MDT0000 complete [ 3627.505876] LustreError: 103137:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3648.991779] Lustre: 101561:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781222146/real 1781222146] req@0000000019179b0a x1867746695449984/t0(0) o400->MGC192.168.201.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781222162 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3649.069524] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3649.110415] LustreError: 102237:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3649.412803] Lustre: server umount lustre-OST0000 complete [ 3665.029248] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 3680.575963] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3726.983427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.343956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3727.375741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3727.510860] Lustre: lustre-MDT0000: new disk, initializing [ 3727.610099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3727.620967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3732.345695] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3744.822313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3745.027889] Lustre: 105919:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3745.103509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3745.120501] Lustre: Skipped 1 previous similar message [ 3745.244822] Lustre: lustre-MDT0001: new disk, initializing [ 3745.449434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3745.503148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3745.523418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3750.623908] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3759.830618] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3764.231393] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3771.621734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3771.864351] Lustre: lustre-OST0000: new disk, initializing [ 3771.866905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3771.871974] Lustre: 107661:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3771.994690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3779.171905] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3780.238614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3780.247649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3780.408539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3788.209847] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3793.284581] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3793.457456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3797.909412] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3798.178899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3801.062638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3801.076114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.079244] LustreError: Skipped 1 previous similar message [ 3801.080123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3801.102908] Lustre: Skipped 1 previous similar message [ 3806.191161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3806.199189] Lustre: Skipped 2 previous similar messages [ 3811.310598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3811.319788] Lustre: Skipped 1 previous similar message [ 3814.368046] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3814.678808] Lustre: server umount lustre-OST0000 complete [ 3821.536816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3821.564808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3821.569412] Lustre: Skipped 1 previous similar message [ 3825.022715] Lustre: server umount lustre-MDT0000 complete [ 3830.673330] LustreError: 105909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222344 with bad export cookie 1868319987159890934 [ 3830.686943] LustreError: 105909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3830.689108] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3831.246774] Lustre: server umount lustre-MDT0001 complete [ 3842.980963] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 19:59:15 (1781222355) [ 3847.325285] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3869.247439] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 3886.200607] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3937.419046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.781307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3937.816686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3937.934745] Lustre: lustre-MDT0000: new disk, initializing [ 3938.063971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3938.080442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3944.297455] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3957.591231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.760719] Lustre: 112410:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3957.905716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3957.923329] Lustre: Skipped 1 previous similar message [ 3958.099940] Lustre: lustre-MDT0001: new disk, initializing [ 3958.278539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3958.349481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3958.408291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3963.580775] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3972.295965] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3976.467966] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3984.966326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3985.305828] Lustre: lustre-OST0000: new disk, initializing [ 3985.309437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3985.315691] Lustre: 114162:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3985.446079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3992.131817] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3993.223701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3993.249276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3993.428972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4001.693817] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4007.282195] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4007.900787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4014.362524] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4014.630635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4019.171019] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.174768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4019.189534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4019.210319] Lustre: Skipped 2 previous similar messages [ 4023.418247] Lustre: server umount lustre-OST0000 complete [ 4029.416505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.417810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4029.431318] Lustre: Skipped 1 previous similar message [ 4029.457240] Lustre: Skipped 2 previous similar messages [ 4034.081628] Lustre: server umount lustre-MDT0000 complete [ 4034.537220] LustreError: 112417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4034.568158] LustreError: 112417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4038.453214] LustreError: 112402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222552 with bad export cookie 1868319987159891816 [ 4038.467589] LustreError: 112402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4038.468423] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4039.046781] Lustre: server umount lustre-MDT0001 complete [ 4048.610752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4049.202784] LustreError: 116232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4049.351961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.486829] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4054.496917] LustreError: 116233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4059.617916] LustreError: 116232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4062.701176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4067.166738] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4073.681740] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4077.946956] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4085.483676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4092.381948] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4098.583399] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4105.967387] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4114.920269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.924925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4114.926599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4114.944023] Lustre: Skipped 1 previous similar message [ 4114.975759] LustreError: Skipped 1 previous similar message [ 4117.055926] Lustre: server umount lustre-OST0000 complete [ 4124.643714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4124.665359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.393079] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4135.838837] Lustre: server umount lustre-MDT0000 complete [ 4140.522811] LustreError: 116227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4140.539771] LustreError: 116227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4140.572211] LustreError: 116213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222654 with bad export cookie 1868319987159892698 [ 4140.586840] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4140.588345] LustreError: 116213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4141.246068] Lustre: server umount lustre-MDT0001 complete [ 4156.732723] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 4160.105483] Key type lgssc unregistered [ 4160.564815] LNet: 119509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4160.572570] LNetError: 119509:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4160.611984] LNet: Removed LNI 192.168.201.122@tcp [ 4161.770086] Key type .llcrypt unregistered [ 4161.772635] Key type ._llcrypt unregistered [ 4182.050333] Key type ._llcrypt registered [ 4182.054853] Key type .llcrypt registered [ 4182.182035] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 4199.865832] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 4201.609484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4201.652134] alg: No test for adler32 (adler32-zlib) [ 4202.856335] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 4203.223428] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 4204.960130] Key type lgssc registered [ 4206.424559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4251.182774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4251.222308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.663587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4252.698556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4252.749178] Lustre: lustre-MDT0000: new disk, initializing [ 4252.824977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.837527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4257.442216] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4270.356074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4270.526978] Lustre: 122631:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4270.591444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4270.596083] Lustre: Skipped 1 previous similar message [ 4270.672846] Lustre: lustre-MDT0001: new disk, initializing [ 4270.780473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4270.834096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4270.849199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4276.009637] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4285.031282] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4289.846777] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4297.807989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4298.062511] Lustre: lustre-OST0000: new disk, initializing [ 4298.066743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4298.070314] Lustre: 124374:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4298.154476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4304.803409] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4305.986482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4305.994894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4306.084895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4313.365993] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4319.077520] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4319.363309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4325.171591] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4325.547095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4332.005908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.005958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4332.021582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4337.131357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4337.136139] Lustre: Skipped 1 previous similar message [ 4342.239220] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4342.245312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4342.256064] Lustre: Skipped 1 previous similar message [ 4342.615367] Lustre: server umount lustre-OST0000 complete [ 4347.367512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.378282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4347.382350] Lustre: Skipped 2 previous similar messages [ 4347.398115] Lustre: Skipped 1 previous similar message [ 4352.481730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4352.489617] Lustre: Skipped 1 previous similar message [ 4353.314318] Lustre: server umount lustre-MDT0000 complete [ 4358.161964] LustreError: 122623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222872 with bad export cookie 13384801401652725010 [ 4358.176657] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4358.186554] LustreError: 122623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4358.760167] Lustre: server umount lustre-MDT0001 complete [ 4370.378798] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 20:08:03 (1781222883) [ 4377.729883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4378.442232] LustreError: 126642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.453666] LustreError: 126642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4378.535440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.874960] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4383.723724] LustreError: 126643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4388.832074] LustreError: 126642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.983204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4394.705696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4400.323122] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4407.603195] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4412.360177] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4419.425480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4419.458234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4419.467839] Lustre: Skipped 1 previous similar message [ 4422.044938] Lustre: server umount lustre-MDT0000 complete [ 4424.680906] LustreError: 127836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4424.704743] LustreError: 127836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4426.227045] LustreError: 126625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222940 with bad export cookie 13384801401652725892 [ 4426.233628] LustreError: 126625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4426.241132] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4426.999162] Lustre: server umount lustre-MDT0001 complete [ 4436.496056] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 20:09:09 (1781222949) [ 4443.797989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4459.423567] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4464.544275] LustreError: 128832:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 4490.271460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4497.563497] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4505.939883] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4508.810099] Lustre: server umount lustre-OST0000 complete [ 4520.943407] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 20:10:34 (1781223034) [ 4529.681664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4530.542681] LustreError: 130166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4530.666416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4536.900522] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4540.896728] LustreError: 130167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.931570] LustreError: 130167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4549.651417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4555.749902] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4563.546229] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4567.913235] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4576.389568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4577.062983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4577.071662] Lustre: Skipped 1 previous similar message [ 4584.503855] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4592.533937] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4611.051147] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.056092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4611.060269] Lustre: Skipped 3 previous similar messages [ 4611.066670] Lustre: Skipped 2 previous similar messages [ 4616.513957] Lustre: server umount lustre-MDT0000 complete [ 4621.188423] LustreError: 130148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223135 with bad export cookie 13384801401652726557 [ 4621.202799] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4621.226887] LustreError: 130148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4621.299873] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.301351] LustreError: 130161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.347873] LustreError: 130161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4622.075138] Lustre: server umount lustre-MDT0001 complete [ 4628.743827] Lustre: server umount lustre-OST0000 complete [ 4638.821148] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 20:12:32 (1781223152) [ 4648.458694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4649.101392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.203898] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4654.570324] LustreError: 133506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4654.586536] LustreError: 133506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4665.223978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.526651] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4678.225348] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4683.760698] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4692.446855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4694.964493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4700.051136] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4707.112640] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4712.799949] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4712.911926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4719.424964] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4719.762243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4725.731616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4725.742668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4725.760789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4725.774650] Lustre: Skipped 5 previous similar messages [ 4728.723162] Lustre: server umount lustre-OST0000 complete [ 4740.476875] Lustre: server umount lustre-MDT0000 complete [ 4741.102987] LustreError: 133507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.126488] LustreError: 133507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4744.895846] LustreError: 133486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223258 with bad export cookie 13384801401652727621 [ 4744.906181] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4745.464644] Lustre: server umount lustre-MDT0001 complete [ 4756.991965] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 20:14:30 (1781223270) [ 4764.662302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4780.319773] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4785.503991] LustreError: 137293:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 4811.423290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.433178] Lustre: Skipped 2 previous similar messages [ 4818.205723] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4825.207257] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4832.719663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4833.536642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4838.151776] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4847.999702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4853.589938] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4860.234594] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4864.294877] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4870.220751] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4870.519727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4875.616910] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4875.840049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4879.329365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4879.337439] LustreError: Skipped 1 previous similar message [ 4879.338977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.353258] Lustre: Skipped 4 previous similar messages [ 4879.355667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4879.378463] Lustre: Skipped 4 previous similar messages [ 4884.457640] LustreError: 138217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4884.478703] LustreError: 138217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4884.603410] Lustre: server umount lustre-MDT0000 complete [ 4889.737196] LustreError: 137301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223403 with bad export cookie 13384801401652728650 [ 4889.743557] LustreError: 137301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4889.747448] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4890.246156] Lustre: server umount lustre-MDT0001 complete [ 4905.215745] Lustre: server umount lustre-OST0000 complete [ 4915.712799] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 20:17:09 (1781223429) [ 4922.605667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4938.207828] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4943.329341] LustreError: 141115:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 4976.300360] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4983.943556] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4991.543112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.216437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4997.095472] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5007.255187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5013.378517] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5020.751692] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5025.204614] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5032.262508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5032.461985] Lustre: lustre-OST0001: new disk, initializing [ 5032.465808] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5032.471443] Lustre: 143649:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5039.256739] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5041.743673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5041.762114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5041.909425] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5047.247374] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5053.197052] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5053.519580] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5058.597476] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5058.900291] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5062.634096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5062.645135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5062.657142] Lustre: Skipped 1 previous similar message [ 5067.977486] Lustre: server umount lustre-OST0000 complete [ 5077.984406] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5088.223752] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5088.690320] Lustre: server umount lustre-OST0001 complete [ 5094.885316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5094.896819] LustreError: Skipped 1 previous similar message [ 5100.884248] Lustre: server umount lustre-MDT0000 complete [ 5105.336233] LustreError: 141122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223619 with bad export cookie 13384801401652729602 [ 5105.349425] LustreError: 141122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5105.363573] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5105.792572] Lustre: server umount lustre-MDT0001 complete [ 5136.148318] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 20:20:49 (1781223649) [ 5138.075437] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5140.514827] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5142.343230] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 20:20:55 (1781223655) [ 5149.962471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5150.484567] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5150.523860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5150.897460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5150.900086] Lustre: Skipped 6 previous similar messages [ 5156.618255] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5166.574498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.684390] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5166.735709] Lustre: 147072:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5172.080822] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5178.774757] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5182.820975] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5192.040978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5192.392846] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5198.068345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5200.041563] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5206.386480] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5212.613613] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5212.924599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5218.061719] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5218.331184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5223.392951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5223.404688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.427830] Lustre: Skipped 4 previous similar messages [ 5226.374359] Lustre: server umount lustre-OST0000 complete [ 5248.881454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5256.201270] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5262.030809] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5264.497531] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.511291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.517822] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5264.540226] Lustre: Skipped 1 previous similar message [ 5268.480563] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5268.711804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5274.159990] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5274.409939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5279.213647] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93999933c800.ost_server_uuid 50 [ 5280.993714] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93999933c800.ost_server_uuid in FULL state after 0 sec [ 5281.267924] LustreError: 147078:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.22@tcp) returned error from blocking AST (req@000000007ca466cd x1867747391973760 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000068f34001/0xb9c05df586a9605c lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.22@tcp remote: 0xc687b61797a895b4 expref: 5 pid: 147078 timeout: 5381 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5281.310789] LustreError: lustre-MDT0000: A client on nid 192.168.201.22@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5281.318076] LustreError: 147067:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.22@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000068f34001/0xb9c05df586a9605c lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.22@tcp remote: 0xc687b61797a895b4 expref: 6 pid: 147078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5286.892037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5286.899033] LustreError: Skipped 1 previous similar message [ 5292.746828] Lustre: server umount lustre-OST0000 complete [ 5302.830390] Lustre: server umount lustre-MDT0000 complete [ 5305.826058] LustreError: 148104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5305.852061] LustreError: 148104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 5306.669037] LustreError: 147064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223820 with bad export cookie 13384801401652730785 [ 5306.681165] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5306.695145] LustreError: 147064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5313.462812] Lustre: server umount lustre-MDT0001 complete [ 5326.160499] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 5328.876866] Key type lgssc unregistered [ 5329.161748] LNet: 152552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5329.169266] LNetError: 152552:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5329.180862] LNet: Removed LNI 192.168.201.122@tcp [ 5330.021131] Key type .llcrypt unregistered [ 5330.023250] Key type ._llcrypt unregistered [ 5353.874783] Key type ._llcrypt registered [ 5353.879318] Key type .llcrypt registered [ 5354.020241] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 5355.644526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5355.802677] alg: No test for adler32 (adler32-zlib) [ 5356.910058] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 5357.294581] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 5359.095133] Key type lgssc registered [ 5360.837890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5370.079634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5370.104621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.804527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5376.623571] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5381.091190] LustreError: 153946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5381.107723] LustreError: 153946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5386.215874] LustreError: 153947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5386.957966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.433726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5393.118173] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5400.123067] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5404.961290] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5412.876353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5413.523948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5416.581320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5421.310985] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5428.448710] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5437.475343] Lustre: Failing over lustre-MDT0000 [ 5438.013974] Lustre: server umount lustre-MDT0000 complete [ 5438.443533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5438.445047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.505362] LustreError: 153947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.515911] LustreError: 153947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5443.557111] LustreError: 153946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5443.578282] LustreError: 153946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5448.673672] LustreError: 153943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5448.695701] LustreError: 153943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5450.749099] LustreError: 153928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.22@tcp arrived at 1781223964 with bad export cookie 13234065587105582085 [ 5454.824255] Lustre: 153259:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781223952/real 1781223952] req@000000003bb1cebf x1867748601784576/t0(0) o400->MGC192.168.201.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781223968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5454.863416] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5459.939275] LustreError: 155155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.971349] LustreError: 155155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5460.592076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.031954] LustreError: 153255:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000091ed6883 x1867748601789824/t0(0) o250->MGC192.168.201.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5464.427719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.473626] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5469.664661] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5469.672724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5470.093285] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5473.050118] Lustre: lustre-MDT0000: Denying connection for new client 620e1f31-7b75-4a6a-bf22-87c8494863e8 (at 192.168.201.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5484.512260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5484.518583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5484.536163] Lustre: Skipped 2 previous similar messages [ 5484.549216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5490.154607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5490.164480] Lustre: Skipped 3 previous similar messages [ 5495.273584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5495.278380] Lustre: Skipped 1 previous similar message [ 5496.287698] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5496.468547] Lustre: server umount lustre-OST0000 complete [ 5500.946368] LustreError: 157324:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5500.955900] Lustre: 156688:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5500.969455] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5500.997418] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5501.008555] LustreError: 156688:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000f76454c x1867748601806720/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5501.029151] LustreError: 156688:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5501.044525] LustreError: 156688:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5501.067100] Lustre: lustre-MDT0000: Recovery over after 0:32, of 2 clients 0 recovered and 2 were evicted. [ 5501.582387] Lustre: server umount lustre-MDT0000 complete [ 5505.533781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5505.548499] Lustre: Skipped 1 previous similar message [ 5505.554235] LustreError: 153942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5505.567454] LustreError: 153942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5505.818150] LustreError: 153928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224019 with bad export cookie 13234065587105582190 [ 5505.821619] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5505.832365] LustreError: 153928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5505.870422] LustreError: 153255:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5508.557322] Lustre: server umount lustre-MDT0001 complete [ 5526.430124] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 5530.137944] Key type lgssc unregistered [ 5530.488389] LNet: 158075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5530.493636] LNetError: 158075:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5530.525156] LNet: Removed LNI 192.168.201.122@tcp [ 5531.348184] Key type .llcrypt unregistered [ 5531.349982] Key type ._llcrypt unregistered [ 5557.586414] Key type ._llcrypt registered [ 5557.595624] Key type .llcrypt registered [ 5557.687200] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 5559.223338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5559.291033] alg: No test for adler32 (adler32-zlib) [ 5560.335943] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 5560.514626] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 5562.167191] Key type lgssc registered [ 5563.470510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5571.956295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5571.992908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.580508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5579.183691] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5583.842226] LustreError: 159469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.856177] LustreError: 159469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5588.972085] LustreError: 159470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5589.194828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5589.880224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5594.836845] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5601.547429] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5606.324075] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5613.816452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5614.057431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5618.162800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5620.115811] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5626.720430] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5635.553400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5635.560814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.570909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5638.628426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5638.636339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5638.636964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5638.656516] Lustre: Skipped 1 previous similar message [ 5641.647916] Lustre: server umount lustre-OST0000 complete [ 5648.864853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.875661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5648.888195] Lustre: Skipped 1 previous similar message [ 5648.908685] Lustre: Skipped 1 previous similar message [ 5652.876640] Lustre: server umount lustre-MDT0000 complete [ 5653.988574] LustreError: 160182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5654.005464] LustreError: 160182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5656.372361] LustreError: 159449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224170 with bad export cookie 4920102043845730901 [ 5656.374820] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5656.382345] LustreError: 159449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5656.716865] Lustre: server umount lustre-MDT0001 complete [ 5667.901444] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 5670.863997] Key type lgssc unregistered [ 5671.203571] LNet: 162645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5671.210479] LNetError: 162645:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5671.233063] LNet: Removed LNI 192.168.201.122@tcp [ 5672.299455] Key type .llcrypt unregistered [ 5672.307147] Key type ._llcrypt unregistered [ 5709.529558] Key type ._llcrypt registered [ 5709.532540] Key type .llcrypt registered [ 5709.648065] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 5711.393469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5711.435609] alg: No test for adler32 (adler32-zlib) [ 5712.630219] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 5712.925643] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 5714.639195] Key type lgssc registered [ 5716.053928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5724.982609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5725.012601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5726.903319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5731.915825] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5736.421570] LustreError: 164333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5736.439725] LustreError: 164333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5741.538483] LustreError: 164332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.306811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.748072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5748.659056] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5754.379549] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5759.131497] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5766.377410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5766.774675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5771.901981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5775.014298] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5783.073916] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5792.740928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5792.742259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5792.749522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5792.778832] Lustre: Skipped 1 previous similar message [ 5797.858871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5797.861399] Lustre: Skipped 1 previous similar message [ 5798.149928] Lustre: server umount lustre-OST0000 complete [ 5802.986076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.990710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5803.003317] Lustre: Skipped 1 previous similar message [ 5803.022441] Lustre: Skipped 1 previous similar message [ 5808.097786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5808.105322] Lustre: Skipped 1 previous similar message [ 5809.062497] Lustre: server umount lustre-MDT0000 complete [ 5813.227104] LustreError: 164333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5813.260253] LustreError: 164333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5814.302226] LustreError: 166774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224328 with bad export cookie 510759905768272936 [ 5814.309693] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5814.326812] LustreError: 166774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5815.254889] Lustre: server umount lustre-MDT0001 complete [ 5828.890878] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 5832.893194] Key type lgssc unregistered [ 5833.143536] LNet: 167511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5833.149863] LNetError: 167511:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5834.217745] LNet: Removed LNI 192.168.201.122@tcp [ 5835.291258] Key type .llcrypt unregistered [ 5835.294456] Key type ._llcrypt unregistered [ 5860.566715] Key type ._llcrypt registered [ 5860.572697] Key type .llcrypt registered [ 5860.780226] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 5862.602078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5862.656654] alg: No test for adler32 (adler32-zlib) [ 5864.250068] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 5864.773512] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 5866.583326] Key type lgssc registered [ 5868.673371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5882.349688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5882.376316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5884.009688] Lustre: *** cfs_fail_loc=135, val=0*** [ 5884.121296] LustreError: 169025:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5884.129351] LustreError: 169025:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.122@tcp: cfg command failed: rc = -2 [ 5884.137699] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5884.137699] [ 5884.141346] LustreError: MGC192.168.201.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5884.166279] LustreError: 168987:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5884.172617] LustreError: 168987:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5884.179351] LustreError: 168987:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5884.184206] LustreError: 168987:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5884.259192] Lustre: server umount lustre-MDT0000 complete [ 5884.260884] LustreError: 168987:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5893.646487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5894.294722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5901.812901] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5904.865677] LustreError: 169396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5904.878309] LustreError: 169396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5909.986137] LustreError: 169397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5912.922399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.545580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5918.286207] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5924.727630] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5929.789497] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5934.055729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5934.064032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5934.064846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5934.071408] Lustre: Skipped 1 previous similar message [ 5938.212780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5938.215679] Lustre: Skipped 1 previous similar message [ 5943.264275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5943.278513] Lustre: Skipped 1 previous similar message [ 5947.359189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5947.516820] Lustre: server umount lustre-MDT0000 complete [ 5948.401245] LustreError: 169393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5948.427707] LustreError: 169393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5950.927798] LustreError: 169377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224464 with bad export cookie 23966086254908886 [ 5950.941022] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5950.945195] LustreError: 169377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5951.246501] Lustre: server umount lustre-MDT0001 complete [ 5964.094688] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 5966.876677] Key type lgssc unregistered [ 5967.124452] LNet: 171479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5967.129242] LNetError: 171479:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5967.149221] LNet: Removed LNI 192.168.201.122@tcp [ 5967.851051] Key type .llcrypt unregistered [ 5967.852596] Key type ._llcrypt unregistered [ 5990.571707] Key type ._llcrypt registered [ 5990.573909] Key type .llcrypt registered [ 5990.664847] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 5990.930696] Key type .llcrypt unregistered [ 5990.933901] Key type ._llcrypt unregistered [ 6010.432912] Key type ._llcrypt registered [ 6010.439343] Key type .llcrypt registered [ 6010.605738] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 6012.112115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6012.153707] alg: No test for adler32 (adler32-zlib) [ 6013.265713] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6013.629327] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 6015.327620] Key type lgssc registered [ 6016.721828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6026.901909] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6026.937196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6042.655473] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6047.775212] LustreError: 173578:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 6074.463418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6082.871518] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6090.501299] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6097.688503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6097.846754] LustreError: 172913:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000dbe7430e x1867749289625856/t0(0) o250->MGC192.168.201.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6098.370415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6098.453023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6102.871682] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6103.526683] LustreError: 173604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.568167] LustreError: 173604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6108.653243] LustreError: 173603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6109.681412] LustreError: 174503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6109.695957] LustreError: 174503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6112.140730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6112.595623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6117.214591] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6124.747382] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6128.510694] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6132.063868] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6148.620692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.624224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6148.677905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6148.709064] Lustre: Skipped 1 previous similar message [ 6151.008933] Lustre: server umount lustre-OST0000 complete [ 6158.833477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.834981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6158.840153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6158.853768] Lustre: Skipped 2 previous similar messages [ 6162.518771] Lustre: server umount lustre-MDT0000 complete [ 6163.936712] LustreError: 174504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.958215] LustreError: 174504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6166.685412] LustreError: 173585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224680 with bad export cookie 2393413185237894492 [ 6166.692777] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6166.700157] LustreError: 173585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6167.064919] Lustre: server umount lustre-MDT0001 complete [ 6180.168521] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 6183.559892] Key type lgssc unregistered [ 6184.138524] LNet: 177189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6184.154809] LNetError: 177189:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6185.196233] LNet: Removed LNI 192.168.201.122@tcp [ 6186.304178] Key type .llcrypt unregistered [ 6186.307841] Key type ._llcrypt unregistered [ 6216.593806] Key type ._llcrypt registered [ 6216.595946] Key type .llcrypt registered [ 6216.734555] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 6218.139915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6218.251041] alg: No test for adler32 (adler32-zlib) [ 6219.426717] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6219.721550] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 6221.391889] Key type lgssc registered [ 6222.513416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6230.753675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6230.777430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6232.607419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6238.446128] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6242.790018] LustreError: 178587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.805208] LustreError: 178587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6247.925676] LustreError: 178586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6249.165212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.383937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6254.313409] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6262.092248] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6267.547436] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6276.479710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6277.019802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6278.067098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6284.580461] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6290.743142] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6298.950961] Lustre: Failing over lustre-MDT0000 [ 6299.626971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6299.697282] LustreError: 178586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6299.723490] LustreError: 178586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6299.829804] Lustre: server umount lustre-MDT0000 complete [ 6303.807878] LustreError: 179648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6308.887836] LustreError: 178582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6308.939340] LustreError: 178582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6315.999931] Lustre: 177897:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781224813/real 1781224813] req@000000006636ce6a x1867749505658240/t0(0) o400->MGC192.168.201.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781224829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6316.009859] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6319.119130] LustreError: 178581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6319.176072] LustreError: 178581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6323.717687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6326.240992] LustreError: 177896:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fad325f7 x1867749505663744/t0(0) o250->MGC192.168.201.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6326.728123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6326.835970] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6329.372102] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6331.796512] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6331.898619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6331.917812] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6331.960135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6337.984846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6342.682701] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6361.067520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6361.078548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6361.098814] Lustre: Skipped 2 previous similar messages [ 6361.108291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6362.593207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6362.624415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6362.631587] Lustre: Skipped 1 previous similar message [ 6364.790527] Lustre: server umount lustre-OST0000 complete [ 6372.834745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6372.847857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6372.861142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6372.911327] Lustre: Skipped 1 previous similar message [ 6375.946231] Lustre: server umount lustre-MDT0000 complete [ 6377.963276] LustreError: 178586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.981489] LustreError: 178586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6380.193353] LustreError: 178566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224894 with bad export cookie 11132733986473686464 [ 6380.209667] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6380.514980] Lustre: server umount lustre-MDT0001 complete [ 6393.612844] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 6397.926651] Key type lgssc unregistered [ 6398.343909] LNet: 183028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6398.351409] LNetError: 183028:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6398.377876] LNet: Removed LNI 192.168.201.122@tcp [ 6399.577886] Key type .llcrypt unregistered [ 6399.582253] Key type ._llcrypt unregistered [ 6426.275878] Key type ._llcrypt registered [ 6426.283778] Key type .llcrypt registered [ 6426.448948] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 6428.422130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6428.464872] alg: No test for adler32 (adler32-zlib) [ 6429.590313] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6429.798292] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 6431.487399] Key type lgssc registered [ 6433.122060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6442.969747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6443.000667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6444.526654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.361151] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6454.767625] LustreError: 184424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6454.801856] LustreError: 184424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6459.878833] LustreError: 184423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6460.236447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.851157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6467.198227] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6474.719678] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6478.546949] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6488.068380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6488.769317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6489.800574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6496.629938] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6502.935945] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6506.232992] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6515.710730] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6535.602634] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6546.400466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6546.412162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6546.428744] LustreError: Skipped 1 previous similar message [ 6546.438520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6546.473152] Lustre: Skipped 1 previous similar message [ 6546.506631] Lustre: Skipped 1 previous similar message [ 6551.540919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6551.548811] Lustre: Skipped 1 previous similar message [ 6552.403619] Lustre: server umount lustre-OST0000 complete [ 6556.642985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6556.665793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6556.670993] Lustre: Skipped 1 previous similar message [ 6561.760891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6561.776702] Lustre: Skipped 1 previous similar message [ 6563.403694] Lustre: server umount lustre-MDT0000 complete [ 6566.896553] LustreError: 184423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6566.920683] LustreError: 184423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6568.110465] LustreError: 184403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225082 with bad export cookie 382111193411303775 [ 6568.117321] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6568.136560] LustreError: 184403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6568.572537] Lustre: server umount lustre-MDT0001 complete [ 6581.665746] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 6585.281210] Key type lgssc unregistered [ 6585.744619] LNet: 187942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6585.753613] LNetError: 187942:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6585.787069] LNet: Removed LNI 192.168.201.122@tcp [ 6586.786348] Key type .llcrypt unregistered [ 6586.788590] Key type ._llcrypt unregistered [ 6614.149588] Key type ._llcrypt registered [ 6614.151678] Key type .llcrypt registered [ 6614.315743] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 6615.617919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6615.663779] alg: No test for adler32 (adler32-zlib) [ 6616.763533] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6617.046221] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 6618.816201] Key type lgssc registered [ 6620.125846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6628.344621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6628.379112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6630.021125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6635.134748] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6639.587657] LustreError: 189192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6639.614444] LustreError: 189192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6644.320048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6644.819731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6650.364799] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6656.931570] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6661.067303] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6668.242269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6668.624826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6674.809539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6675.853478] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6684.224114] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6692.267262] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6707.120839] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6720.330623] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6729.001324] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6741.475062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6741.487092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6741.510097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6741.520287] Lustre: Skipped 1 previous similar message [ 6746.532190] Lustre: server umount lustre-OST0000 complete [ 6751.714726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6751.730275] Lustre: Skipped 1 previous similar message [ 6751.738391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6751.748472] Lustre: Skipped 1 previous similar message [ 6756.834195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6756.848260] Lustre: Skipped 1 previous similar message [ 6757.398614] Lustre: server umount lustre-MDT0000 complete [ 6761.961613] LustreError: 190401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6761.976380] LustreError: 190401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6762.183197] LustreError: 189172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225276 with bad export cookie 14858480767900251029 [ 6762.185952] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6762.192517] LustreError: 189172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6762.724841] Lustre: server umount lustre-MDT0001 complete [ 6777.988025] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 6781.247173] Key type lgssc unregistered [ 6781.529686] LNet: 193909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6781.533992] LNetError: 193909:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6781.546205] LNet: Removed LNI 192.168.201.122@tcp [ 6782.221695] Key type .llcrypt unregistered [ 6782.224161] Key type ._llcrypt unregistered [ 6809.423637] Key type ._llcrypt registered [ 6809.431496] Key type .llcrypt registered [ 6809.633471] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 6811.771303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6811.872338] alg: No test for adler32 (adler32-zlib) [ 6813.100461] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6813.433908] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 6815.255211] Key type lgssc registered [ 6816.674920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6825.995259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6826.008097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6827.577734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6832.897295] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6837.734503] LustreError: 195308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6837.750368] LustreError: 195308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6841.920440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6842.429828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6847.726154] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6855.340421] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6860.375379] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6868.712222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6869.318220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6872.452713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6877.746607] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6885.095986] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6903.265689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6903.275384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6903.288979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6903.330990] Lustre: Skipped 1 previous similar message [ 6908.062524] Lustre: server umount lustre-OST0000 complete [ 6913.504092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6913.511620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6913.531855] Lustre: Skipped 1 previous similar message [ 6913.547078] Lustre: Skipped 2 previous similar messages [ 6918.635955] LustreError: 195308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6918.775925] Lustre: server umount lustre-MDT0000 complete [ 6923.168466] LustreError: 195287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225437 with bad export cookie 460867429552710380 [ 6923.170290] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6923.179128] LustreError: 195287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6923.633710] Lustre: server umount lustre-MDT0001 complete [ 6935.785681] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 6938.915947] Key type lgssc unregistered [ 6939.321446] LNet: 198634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6939.331034] LNetError: 198634:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6939.366783] LNet: Removed LNI 192.168.201.122@tcp [ 6940.620747] Key type .llcrypt unregistered [ 6940.623516] Key type ._llcrypt unregistered [ 6963.028355] Key type ._llcrypt registered [ 6963.030883] Key type .llcrypt registered [ 6963.183222] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 6964.207806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6964.377621] alg: No test for adler32 (adler32-zlib) [ 6966.057920] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6966.424288] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 6968.119191] Key type lgssc registered [ 6969.601831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6977.454729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6977.468444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6979.000502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6984.624255] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6988.257597] LustreError: 199886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.290506] LustreError: 199886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6993.382668] LustreError: 199887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.588360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.044091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7000.808381] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7007.550236] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7013.014920] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7022.379546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7023.270870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7024.334429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7032.210804] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7039.731018] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7060.450472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7060.455208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.459116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7060.461462] LustreError: Skipped 1 previous similar message [ 7060.500777] Lustre: Skipped 1 previous similar message [ 7062.418648] Lustre: server umount lustre-OST0000 complete [ 7070.690249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7070.694126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7070.708542] Lustre: Skipped 1 previous similar message [ 7070.718974] Lustre: Skipped 2 previous similar messages [ 7075.136724] Lustre: server umount lustre-MDT0000 complete [ 7075.810309] LustreError: 200597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7075.830449] LustreError: 200597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7079.862389] LustreError: 199866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225593 with bad export cookie 8525450493238197535 [ 7079.879678] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7079.880497] LustreError: 199866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7080.548791] Lustre: server umount lustre-MDT0001 complete [ 7093.997761] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 7097.063888] Key type lgssc unregistered [ 7097.406761] LNet: 203214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7097.430115] LNetError: 203214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7097.453069] LNet: Removed LNI 192.168.201.122@tcp [ 7098.550297] Key type .llcrypt unregistered [ 7098.551957] Key type ._llcrypt unregistered [ 7120.679562] Key type ._llcrypt registered [ 7120.686729] Key type .llcrypt registered [ 7120.833711] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7122.284427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7122.376273] alg: No test for adler32 (adler32-zlib) [ 7123.500461] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 7123.740745] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 7125.426132] Key type lgssc registered [ 7126.656919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7136.055301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7136.089633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.762587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7144.378757] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7148.007323] LustreError: 204464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7148.027502] LustreError: 204464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7153.131806] LustreError: 204465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.579362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.158085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7161.077946] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7168.454408] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7173.549405] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7182.803603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7183.337980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7184.398609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7190.319854] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7197.263878] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7207.899602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7207.975374] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7208.340520] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7215.838062] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7222.351833] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7224.243476] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7236.075769] Lustre: Permanently deactivating lustre-OST0001 [ 7236.091992] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7242.368192] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7251.161200] Lustre: Permanently reactivating lustre-OST0001 [ 7251.191498] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7251.206776] Lustre: Skipped 2 previous similar messages [ 7261.228884] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7261.249257] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7261.255819] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7261.268310] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7261.320616] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7263.209173] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7263.221390] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7263.239152] Lustre: Skipped 2 previous similar messages [ 7263.265337] Lustre: Skipped 1 previous similar message [ 7264.358529] Lustre: server umount lustre-OST0001 complete [ 7268.325443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7268.328099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7268.329357] LustreError: 206061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7268.331988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7268.344385] Lustre: Skipped 1 previous similar message [ 7268.362918] Lustre: Skipped 1 previous similar message [ 7273.447884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7273.451198] LustreError: 206061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7273.495042] LustreError: 206061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7274.509209] Lustre: server umount lustre-OST0000 complete [ 7283.680391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.694926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7283.702314] Lustre: Skipped 2 previous similar messages [ 7285.562881] Lustre: server umount lustre-MDT0000 complete [ 7288.810755] LustreError: 206014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7288.848603] LustreError: 206014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7289.761477] LustreError: 204446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225803 with bad export cookie 16886001422161879739 [ 7289.765221] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7289.769534] LustreError: 204446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7290.100544] Lustre: server umount lustre-MDT0001 complete [ 7306.296354] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 7311.183587] Key type lgssc unregistered [ 7311.752137] LNet: 209042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7311.762900] LNetError: 209042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7312.813344] LNet: Removed LNI 192.168.201.122@tcp [ 7314.436927] Key type .llcrypt unregistered [ 7314.443212] Key type ._llcrypt unregistered [ 7365.035724] Key type ._llcrypt registered [ 7365.037288] Key type .llcrypt registered [ 7365.144432] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7367.302583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7367.322740] alg: No test for adler32 (adler32-zlib) [ 7368.392598] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 7368.686922] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 7370.457106] Key type lgssc registered [ 7371.509826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7380.259734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7380.275603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7381.637678] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7381.668573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7381.915813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.075823] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7397.477668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7397.578160] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7397.612522] Lustre: 211235:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7397.846077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7402.110812] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7408.549566] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7413.957095] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7421.461732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7421.618493] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7421.964877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7428.164219] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7430.843161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7434.889584] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7439.297329] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7446.429342] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7454.323788] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7462.640883] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7472.076453] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7497.876149] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7497.888423] Lustre: Skipped 2 previous similar messages [ 7536.647876] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7536.651497] Lustre: Skipped 4 previous similar messages [ 7604.973626] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7604.982529] Lustre: Skipped 7 previous similar messages [ 7620.067172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7620.079887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7620.106458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7620.581539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7620.609096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7620.613136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7625.702681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7625.708790] Lustre: Skipped 1 previous similar message [ 7626.211974] Lustre: server umount lustre-OST0000 complete [ 7630.831833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.841407] Lustre: Skipped 1 previous similar message [ 7630.842396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7630.851172] Lustre: Skipped 1 previous similar message [ 7635.937593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7635.951672] Lustre: Skipped 1 previous similar message [ 7636.592965] Lustre: server umount lustre-MDT0000 complete [ 7640.471631] LustreError: 214245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226154 with bad export cookie 1796408460374779893 [ 7640.475833] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7640.487285] LustreError: 214245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7640.881596] Lustre: server umount lustre-MDT0001 complete [ 7653.151847] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 7657.593892] Key type lgssc unregistered [ 7658.092196] LNet: 215667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7658.102458] LNetError: 215667:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7658.142841] LNet: Removed LNI 192.168.201.122@tcp [ 7659.904149] Key type .llcrypt unregistered [ 7659.916269] Key type ._llcrypt unregistered [ 7690.388148] Key type ._llcrypt registered [ 7690.399623] Key type .llcrypt registered [ 7690.675634] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7691.866499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7691.893849] alg: No test for adler32 (adler32-zlib) [ 7692.912520] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 7693.238387] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 7694.992050] Key type lgssc registered [ 7696.274932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7704.329528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7704.350554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7705.650461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7710.785720] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7715.812497] LustreError: 217062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7715.835418] LustreError: 217062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7719.306249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7719.642295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7724.451528] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7730.387864] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7734.891781] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7741.663612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7742.230407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7748.684430] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7750.390219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7754.276523] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7773.784566] Lustre: server umount lustre-OST0000 complete [ 7781.345394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7781.348186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7781.351763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7781.372051] Lustre: Skipped 1 previous similar message [ 7781.382579] Lustre: Skipped 1 previous similar message [ 7786.468514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7786.473081] Lustre: Skipped 1 previous similar message [ 7787.563601] Lustre: server umount lustre-MDT0000 complete [ 7791.586233] Lustre: 216374:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226289/real 1781226289] req@00000000fc9f606d x1867751051267584/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781226305 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7791.597717] LustreError: 217063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7791.634946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7791.689154] LustreError: 217063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7791.869620] LustreError: 219302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226305 with bad export cookie 118813096182139066 [ 7791.871255] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7791.882018] LustreError: 219302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7792.147457] Lustre: server umount lustre-MDT0001 complete [ 7805.216299] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 7808.370680] Key type lgssc unregistered [ 7808.694825] LNet: 220394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7808.706369] LNetError: 220394:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7808.721845] LNet: Removed LNI 192.168.201.122@tcp [ 7809.922461] Key type .llcrypt unregistered [ 7809.924438] Key type ._llcrypt unregistered [ 7835.591046] Key type ._llcrypt registered [ 7835.592265] Key type .llcrypt registered [ 7835.669530] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7836.983971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7837.003114] alg: No test for adler32 (adler32-zlib) [ 7838.036268] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 7838.138935] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 7839.871229] Key type lgssc registered [ 7841.706564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8114.232968] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 8118.216971] Key type lgssc unregistered [ 8118.897390] LNet: 222383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8118.924824] LNetError: 222383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8118.972334] LNet: Removed LNI 192.168.201.122@tcp [ 8120.336538] Key type .llcrypt unregistered [ 8120.351415] Key type ._llcrypt unregistered [ 8154.189708] Key type ._llcrypt registered [ 8154.191904] Key type .llcrypt registered [ 8154.279976] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 8171.049275] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 8172.568795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8172.626345] alg: No test for adler32 (adler32-zlib) [ 8173.757726] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 8174.103967] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 8175.743194] Key type lgssc registered [ 8177.202130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8216.744849] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7925 sec ========== 21:12:09 (1781226729) [ 8219.161824] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:12:11 (1781226731) === [ 8223.415570] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:12:16 (1781226736) === [ 8270.840890] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 8272.688920] Key type lgssc unregistered [ 8272.858956] LNet: 227570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8272.862938] LNetError: 227570:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8272.876363] LNet: Removed LNI 192.168.201.122@tcp [ 8273.336559] Key type .llcrypt unregistered [ 8273.337841] Key type ._llcrypt unregistered