[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488494395 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003112] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.006011] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.011173] LSM: Security Framework initializing [ 0.013058] Yama: becoming mindful. [ 0.014062] SELinux: Initializing. [ 0.015077] *** VALIDATE selinux *** [ 0.024857] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030711] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031194] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032132] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034073] *** VALIDATE tmpfs *** [ 0.036300] *** VALIDATE proc *** [ 0.037343] *** VALIDATE cgroup *** [ 0.038016] *** VALIDATE cgroup2 *** [ 0.040175] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042051] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044039] Spectre V2 : User space: Vulnerable [ 0.045013] Speculative Store Bypass: Vulnerable [ 0.048116] debug: unmapping init [mem 0xffffffff97859000-0xffffffff97860fff] [ 0.050967] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051700] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052026] ... version: 2 [ 0.053013] ... bit width: 48 [ 0.054011] ... generic registers: 4 [ 0.055013] ... value mask: 0000ffffffffffff [ 0.056015] ... max period: 00007fffffffffff [ 0.057013] ... fixed-purpose events: 3 [ 0.058012] ... event mask: 000000070000000f [ 0.060219] rcu: Hierarchical SRCU implementation. [ 0.062499] smp: Bringing up secondary CPUs ... [ 0.063525] x86: Booting SMP configuration: [ 0.064030] .... node #0, CPUs: #1 #2 #3 [ 0.068013] smp: Brought up 1 node, 4 CPUs [ 0.070021] smpboot: Max logical packages: 1 [ 0.071027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.241472] node 0 deferred pages initialised in 168ms [ 0.244490] devtmpfs: initialized [ 0.246350] x86/mm: Memory block size: 128MB [ 0.249051] gcov: version magic: 0x41383552 [ 0.252396] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.255154] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.257409] pinctrl core: initialized pinctrl subsystem [ 0.259243] [ 0.259658] ************************************************************* [ 0.260011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262014] ** ** [ 0.264015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.266016] ** ** [ 0.267013] ** This means that this kernel is built to expose internal ** [ 0.269014] ** IOMMU data structures, which may compromise security on ** [ 0.271012] ** your system. ** [ 0.273016] ** ** [ 0.275016] ** If you see this message and you are not debugging the ** [ 0.277016] ** kernel, report this immediately to your vendor! ** [ 0.279017] ** ** [ 0.282018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.284017] ************************************************************* [ 0.286821] NET: Registered protocol family 16 [ 0.289564] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.292102] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.295101] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.298146] cpuidle: using governor menu [ 0.299624] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.302559] PCI: Using configuration type 1 for base access [ 0.304166] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.315055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.317088] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.321109] cryptd: max_cpu_qlen set to 1000 [ 0.325028] ACPI: Added _OSI(Module Device) [ 0.326020] ACPI: Added _OSI(Processor Device) [ 0.328024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.330018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.334903] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.340625] ACPI: Interpreter enabled [ 0.342075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.344019] ACPI: Using IOAPIC for interrupt routing [ 0.345159] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.349459] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.359400] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.361055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.364033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.368136] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.372358] acpiphp: Slot [2] registered [ 0.373140] acpiphp: Slot [3] registered [ 0.374113] acpiphp: Slot [4] registered [ 0.375162] acpiphp: Slot [5] registered [ 0.377140] acpiphp: Slot [6] registered [ 0.378135] acpiphp: Slot [7] registered [ 0.380159] acpiphp: Slot [8] registered [ 0.381193] acpiphp: Slot [9] registered [ 0.383158] acpiphp: Slot [10] registered [ 0.384000] acpiphp: Slot [11] registered [ 0.384000] acpiphp: Slot [12] registered [ 0.384122] acpiphp: Slot [13] registered [ 0.385000] acpiphp: Slot [14] registered [ 0.387118] acpiphp: Slot [15] registered [ 0.389133] acpiphp: Slot [16] registered [ 0.390142] acpiphp: Slot [17] registered [ 0.392176] acpiphp: Slot [18] registered [ 0.393134] acpiphp: Slot [19] registered [ 0.395107] acpiphp: Slot [20] registered [ 0.396127] acpiphp: Slot [21] registered [ 0.398093] acpiphp: Slot [22] registered [ 0.399127] acpiphp: Slot [23] registered [ 0.401081] acpiphp: Slot [24] registered [ 0.402095] acpiphp: Slot [25] registered [ 0.403089] acpiphp: Slot [26] registered [ 0.405156] acpiphp: Slot [27] registered [ 0.406095] acpiphp: Slot [28] registered [ 0.408137] acpiphp: Slot [29] registered [ 0.409102] acpiphp: Slot [30] registered [ 0.411116] acpiphp: Slot [31] registered [ 0.412069] PCI host bridge to bus 0000:00 [ 0.414024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.416033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.418025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.420026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.423027] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.425038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.427230] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.432274] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.437675] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.447021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.452763] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.455029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.457035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.460025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.462287] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.464832] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.467059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.469746] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.475030] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.488018] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.495195] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.500766] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.506025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.512026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.528030] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.540968] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.547029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.555023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.573028] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.584062] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.591021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.595021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.612042] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.623047] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.628021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.635020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.649024] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.659352] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.665137] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.670019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.685022] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.694172] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.701023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.708024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.720028] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.733499] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.735449] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.738422] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.740432] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.743308] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.748144] iommu: Default domain type: Passthrough [ 0.750553] SCSI subsystem initialized [ 0.752190] ACPI: bus type USB registered [ 0.753108] usbcore: registered new interface driver usbfs [ 0.755150] usbcore: registered new interface driver hub [ 0.757106] usbcore: registered new device driver usb [ 0.759185] pps_core: LinuxPPS API ver. 1 registered [ 0.761016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.764092] PTP clock support registered [ 0.766116] EDAC MC: Ver: 3.0.0 [ 0.768218] PCI: Using ACPI for IRQ routing [ 0.769627] NetLabel: Initializing [ 0.771015] NetLabel: domain hash size = 128 [ 0.772015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.774086] NetLabel: unlabeled traffic allowed by default [ 0.775145] vgaarb: loaded [ 0.777320] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.779020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.784007] clocksource: Switched to clocksource kvm-clock [ 0.894094] VFS: Disk quotas dquot_6.6.0 [ 0.896270] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.898472] *** VALIDATE ramfs *** [ 0.899850] *** VALIDATE hugetlbfs *** [ 0.901612] pnp: PnP ACPI init [ 0.904090] pnp: PnP ACPI: found 6 devices [ 0.922668] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.925881] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.928185] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.929806] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.932474] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.934853] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.937774] NET: Registered protocol family 2 [ 0.940382] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.945129] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.948021] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.954393] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.958929] TCP: Hash tables configured (established 65536 bind 65536) [ 0.962309] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.966164] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.968923] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971522] NET: Registered protocol family 1 [ 0.974280] RPC: Registered named UNIX socket transport module. [ 0.976405] RPC: Registered udp transport module. [ 0.978789] RPC: Registered tcp transport module. [ 0.980835] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.983268] NET: Registered protocol family 44 [ 0.984933] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.987386] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.989695] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.992348] PCI: CLS 0 bytes, default 64 [ 0.994180] Unpacking initramfs... [ 2.445111] debug: unmapping init [mem 0xffff9068fcc54000-0xffff9068fffbffff] [ 2.450088] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.452169] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.456134] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.998899] Initialise system trusted keyrings [ 3.000819] Key type blacklist registered [ 3.004322] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.014825] zbud: loaded [ 3.018154] *** VALIDATE nfs *** [ 3.019368] *** VALIDATE nfs4 *** [ 3.021235] pstore: using deflate compression [ 3.025914] Platform Keyring initialized [ 3.169737] NET: Registered protocol family 38 [ 3.171850] Key type asymmetric registered [ 3.173537] Asymmetric key parser 'x509' registered [ 3.175500] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.178715] io scheduler mq-deadline registered [ 3.180361] io scheduler kyber registered [ 3.181817] io scheduler bfq registered [ 3.183742] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.186598] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.189380] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.192302] ACPI: Power Button [PWRF] [ 3.284394] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.375289] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.566275] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.676399] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.877864] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.912395] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.948046] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.953850] Non-volatile memory driver v1.3 [ 3.955558] Linux agpgart interface v0.103 [ 3.994600] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.998280] vda: detected capacity change from 0 to 68530176 [ 4.016050] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.019164] vdb: detected capacity change from 0 to 1073741824 [ 4.037328] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.040184] vdc: detected capacity change from 0 to 2621440000 [ 4.053710] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.056829] vdd: detected capacity change from 0 to 2621440000 [ 4.071138] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.074424] vde: detected capacity change from 0 to 4294967296 [ 4.089969] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.093076] vdf: detected capacity change from 0 to 4294967296 [ 4.101385] libphy: Fixed MDIO Bus: probed [ 4.106516] usbcore: registered new interface driver usbserial_generic [ 4.108952] usbserial: USB Serial support registered for generic [ 4.111345] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.116447] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.118418] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.120955] mousedev: PS/2 mouse device common for all mice [ 4.123792] rtc_cmos 00:05: RTC can wake from S4 [ 4.126908] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.127191] rtc_cmos 00:05: registered as rtc0 [ 4.131344] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.135889] intel_pstate: CPU model not supported [ 4.138647] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.142392] hid: raw HID events driver (C) Jiri Kosina [ 4.142396] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.149701] usbcore: registered new interface driver usbhid [ 4.151744] usbhid: USB HID core driver [ 4.153449] drop_monitor: Initializing network drop monitor service [ 4.155652] Initializing XFRM netlink socket [ 4.157819] NET: Registered protocol family 10 [ 4.161488] Segment Routing with IPv6 [ 4.163063] NET: Registered protocol family 17 [ 4.165272] mpls_gso: MPLS GSO support [ 4.170501] RAS: Correctable Errors collector initialized. [ 4.177992] AVX version of gcm_enc/dec engaged. [ 4.179692] AES CTR mode by8 optimization enabled [ 4.277308] sched_clock: Marking stable (4277262190, 0)->(5158838214, -881576024) [ 4.280602] registered taskstats version 1 [ 4.282918] Loading compiled-in X.509 certificates [ 4.286626] zswap: loaded using pool lzo/zbud [ 4.315068] Key type big_key registered [ 4.331284] Key type encrypted registered [ 4.333123] ima: No TPM chip found, activating TPM-bypass! [ 4.334991] ima: Allocated hash algorithm: sha1 [ 4.336781] ima: No architecture policies found [ 4.338578] evm: Initialising EVM extended attributes: [ 4.340623] evm: security.selinux [ 4.341663] evm: security.ima [ 4.342708] evm: security.capability [ 4.346518] evm: HMAC attrs: 0x1 [ 4.349511] rtc_cmos 00:05: setting system clock to 2025-11-16 22:52:47 UTC (1763333567) [ 4.356050] debug: unmapping init [mem 0xffffffff98803000-0xffffffff989fffff] [ 4.359170] debug: unmapping init [mem 0xffffffff97582000-0xffffffff97858fff] [ 4.370146] Write protecting the kernel read-only data: 28672k [ 4.373411] debug: unmapping init [mem 0xffffffff95c03000-0xffffffff95dfffff] [ 4.376181] debug: unmapping init [mem 0xffffffff96514000-0xffffffff965fffff] [ 4.413059] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.423237] systemd[1]: Detected virtualization kvm. [ 4.425330] systemd[1]: Detected architecture x86-64. [ 4.427651] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.453409] systemd[1]: No hostname configured. [ 4.455267] systemd[1]: Set hostname to . [ 4.457675] random: systemd: uninitialized urandom read (16 bytes read) [ 4.460388] systemd[1]: Initializing machine ID from random generator. [ 4.511673] random: ln: uninitialized urandom read (6 bytes read) [ 4.602240] random: systemd: uninitialized urandom read (16 bytes read) [ 4.604879] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.610372] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.616346] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.312746] device-mapper: uevent: version 1.0.3 [ 5.315204] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.997680] virtio_net virtio0 ens2: renamed from eth0 [ 6.022032] scsi host0: ata_piix [ 6.033242] scsi host1: ata_piix [ 6.034809] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.039426] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 6.072798] random: fast init done [ 10.461496] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.855471] random: crng init done [ 10.856766] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.286725] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.547324] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.835422] SELinux: Disabled at runtime. [ 12.896405] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.904802] systemd[1]: Detected virtualization kvm. [ 12.906707] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.467392] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.470829] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.476318] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.481502] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.486253] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.495942] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.507148] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ 13.633431] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 14.036696] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.436038] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.483618] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.613255] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.630261] EDAC sbridge: Ver: 1.1.2 [ 16.386838] Key type dns_resolver registered [ 16.706333] NFS: Registering the id_resolver key type [ 16.708426] Key type id_resolver registered [ 16.709797] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg243-server login: [ 57.095028] hrtimer: interrupt took 3668812 ns [ 61.126133] libcfs: loading out-of-tree module taints kernel. [ 61.153956] Key type ._llcrypt registered [ 61.159680] Key type .llcrypt registered [ 61.258206] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 76.212960] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 77.874340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 77.904027] alg: No test for adler32 (adler32-zlib) [ 79.346689] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 80.243735] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 81.959225] Key type lgssc registered [ 83.392842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.188357] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 96.793835] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.603801] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.992062] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.895621] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.870343] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 132.870460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 132.920905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 132.947271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.166420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 134.196114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 134.281181] Lustre: lustre-MDT0000: new disk, initializing [ 134.399129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.412587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 138.053593] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 150.327694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 150.385823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 150.479081] Lustre: 6507:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 150.501083] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 150.515785] Lustre: Skipped 1 previous similar message [ 150.607927] Lustre: lustre-MDT0001: new disk, initializing [ 150.685864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 150.734606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 150.737745] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 154.584949] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.204791] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 167.181882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 167.286259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 167.443478] Lustre: lustre-OST0000: new disk, initializing [ 167.446549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 167.520250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.969126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 170.974697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 171.064850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 172.723288] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.918290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.029418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 188.092408] Lustre: lustre-OST0001: new disk, initializing [ 188.096861] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 188.162995] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 194.116975] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.144298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 197.150104] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 197.219057] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 206.733947] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 214.496257] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 225.099860] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing check_logdir /tmp/testlogs/ [ 230.038678] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing yml_node [ 233.595881] Lustre: DEBUG MARKER: Client: 2.16.61.54 [ 235.851618] Lustre: DEBUG MARKER: MDS: 2.16.61.54 [ 238.149127] Lustre: DEBUG MARKER: OSS: 2.16.61.54 [ 239.650922] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 17:56:42 EST 2025 [ 254.257318] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 255.366130] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 273.890799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.892573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 273.899945] Lustre: Skipped 2 previous similar messages [ 273.911245] Lustre: Skipped 1 previous similar message [ 275.390435] LustreError: 12209:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 275.458140] Lustre: server umount lustre-MDT0000 complete [ 278.455062] LustreError: 6498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333841 with bad export cookie 14569404305080416096 [ 278.455489] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 278.472718] LustreError: 6498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 279.009728] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.010581] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 279.021820] Lustre: Skipped 2 previous similar messages [ 279.039091] Lustre: Skipped 1 previous similar message [ 283.041217] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 283.044841] Lustre: Skipped 1 previous similar message [ 284.629897] LustreError: 12410:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 284.638229] LustreError: 12410:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 284.750189] Lustre: server umount lustre-MDT0001 complete [ 287.522132] LustreError: 12612:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 287.533176] LustreError: 12612:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 287.565512] Lustre: server umount lustre-OST0000 complete [ 290.279433] LustreError: 12812:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 290.282718] LustreError: 12812:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 290.379393] Lustre: server umount lustre-OST0001 complete [ 294.891577] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 301.116963] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 309.828512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 315.055811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 320.467698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 326.467119] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.457934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.506671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.711580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 332.745831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 332.801677] Lustre: lustre-MDT0000: new disk, initializing [ 332.904221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 332.925929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 336.261309] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 344.232741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 344.300858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 344.393220] Lustre: 14995:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 344.445597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 344.449206] Lustre: Skipped 1 previous similar message [ 344.503232] Lustre: lustre-MDT0001: new disk, initializing [ 344.567523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 344.601426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 344.606334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 347.751924] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 354.704863] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 357.820958] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 363.987941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 364.052347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 364.245988] Lustre: lustre-OST0000: new disk, initializing [ 364.248225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 364.348448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 365.943939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 365.952809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 366.077190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 368.998205] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 375.926622] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 380.188333] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 380.436765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 384.320657] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 384.591276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 386.528242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 386.531898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.544560] Lustre: Skipped 1 previous similar message [ 386.544964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 386.554907] Lustre: Skipped 1 previous similar message [ 391.674840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 391.680261] Lustre: Skipped 1 previous similar message [ 392.535453] LustreError: 17939:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 392.540857] LustreError: 17939:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 392.798147] Lustre: server umount lustre-OST0000 complete [ 396.772721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.892670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 401.899792] Lustre: Skipped 3 previous similar messages [ 402.735987] LustreError: 18141:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 402.742723] LustreError: 18141:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 402.876501] Lustre: server umount lustre-MDT0000 complete [ 406.237552] LustreError: 16745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333969 with bad export cookie 14569404305080417629 [ 406.245325] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 406.248079] LustreError: 16745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 406.608229] Lustre: server umount lustre-MDT0001 complete [ 409.864468] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:59:32 (1763333972) [ 415.546312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.921538] LustreError: 18870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.943361] LustreError: 18870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 415.998615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.959819] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 421.346441] LustreError: 18870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 425.826586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 429.138974] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 433.934836] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 437.617726] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 444.318452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 449.292526] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 453.994662] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 472.548555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 472.550544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.557041] LustreError: Skipped 1 previous similar message [ 472.583479] Lustre: Skipped 1 previous similar message [ 472.591093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 472.593059] Lustre: Skipped 1 previous similar message [ 476.678164] LustreError: 21137:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 476.682269] LustreError: 21137:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 476.796628] Lustre: server umount lustre-OST0000 complete [ 482.272169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 482.281050] LustreError: Skipped 1 previous similar message [ 482.295044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.308222] Lustre: Skipped 1 previous similar message [ 486.205875] Lustre: server umount lustre-MDT0000 complete [ 487.907621] LustreError: 18870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.931824] LustreError: 18870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 489.533772] LustreError: 18850:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334052 with bad export cookie 14569404305080418511 [ 489.539203] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.553366] LustreError: 18850:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 489.893432] Lustre: server umount lustre-MDT0001 complete [ 499.486546] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 502.448621] Key type lgssc unregistered [ 502.808600] LNet: 22087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.832636] LNetError: 22087:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 502.857223] LNet: Removed LNI 192.168.202.143@tcp [ 503.847154] Key type .llcrypt unregistered [ 503.848752] Key type ._llcrypt unregistered [ 527.287327] Key type ._llcrypt registered [ 527.289092] Key type .llcrypt registered [ 527.391870] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 528.408212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 528.520728] alg: No test for adler32 (adler32-zlib) [ 529.834467] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 530.065995] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 531.800720] Key type lgssc registered [ 532.996677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 540.228952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 540.258863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.829738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 545.614243] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 551.394960] LustreError: 23479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.414709] LustreError: 23479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 553.108392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 553.546309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 556.392823] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 560.120511] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 562.919182] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 568.526615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 568.911766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.220876] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 577.642468] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 586.365296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 596.974593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.981852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 596.982517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 596.984968] Lustre: Skipped 1 previous similar message [ 600.571899] LustreError: 26165:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 600.676131] Lustre: server umount lustre-OST0000 complete [ 604.642272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 604.649822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.671989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 604.685764] Lustre: Skipped 1 previous similar message [ 607.203364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 607.208411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.676843] LustreError: 26367:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 610.679287] LustreError: 26367:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 610.807067] Lustre: server umount lustre-MDT0000 complete [ 612.342584] LustreError: 23473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.366694] LustreError: 23473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 614.113267] LustreError: 23458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334177 with bad export cookie 16827725693062542776 [ 614.115298] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 614.143934] LustreError: 23458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 614.249721] LustreError: 26568:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 614.254033] LustreError: 26568:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 614.477549] Lustre: server umount lustre-MDT0001 complete [ 624.144315] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 626.142690] Key type lgssc unregistered [ 626.385268] LNet: 27101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.394889] LNetError: 27101:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.412554] LNet: Removed LNI 192.168.202.143@tcp [ 627.015472] Key type .llcrypt unregistered [ 627.017755] Key type ._llcrypt unregistered [ 646.000967] Key type ._llcrypt registered [ 646.002627] Key type .llcrypt registered [ 646.102698] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 646.787991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 646.795427] alg: No test for adler32 (adler32-zlib) [ 647.823959] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 648.012868] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 649.655158] Key type lgssc registered [ 650.603166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 658.025886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 658.041895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.560438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.613367] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 669.127261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 669.155718] LustreError: 28473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 669.168611] LustreError: 28473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 669.403780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 672.313371] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 676.269964] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 678.999686] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 689.524332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 689.825067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 694.493887] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 699.708939] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 704.191135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 711.881493] LustreError: 31068:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 711.999751] Lustre: server umount lustre-OST0000 complete [ 719.841093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.849403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 719.868882] Lustre: Skipped 1 previous similar message [ 724.966771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 724.969460] Lustre: Skipped 2 previous similar messages [ 726.005147] LustreError: 31269:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 726.010215] LustreError: 31269:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 726.090111] Lustre: server umount lustre-MDT0000 complete [ 730.030943] LustreError: 28454:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334293 with bad export cookie 3828219151785438933 [ 730.034566] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 730.041204] LustreError: 28454:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 740.338383] LustreError: 31472:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 740.352221] LustreError: 31472:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 740.546655] Lustre: server umount lustre-MDT0001 complete [ 750.608364] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 752.731426] Key type lgssc unregistered [ 752.938393] LNet: 32006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.943482] LNetError: 32006:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 752.952942] LNet: Removed LNI 192.168.202.143@tcp [ 753.554312] Key type .llcrypt unregistered [ 753.558088] Key type ._llcrypt unregistered [ 772.672501] Key type ._llcrypt registered [ 772.674814] Key type .llcrypt registered [ 772.777703] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 773.537717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 773.621959] alg: No test for adler32 (adler32-zlib) [ 774.709211] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 774.920117] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 776.583171] Key type lgssc registered [ 777.774323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 784.620512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 784.643660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.223969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.862381] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 796.644429] LustreError: 33378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 796.656838] LustreError: 33378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 797.352104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 797.671522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 801.049272] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 805.780440] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 809.360549] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 815.332389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 815.609646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.162344] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 825.218259] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 830.968763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 838.115557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 838.127203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.143678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 841.191924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 841.193292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 841.193802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.204322] Lustre: Skipped 1 previous similar message [ 843.521105] LustreError: 35646:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 843.655725] Lustre: server umount lustre-OST0000 complete [ 848.865658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 848.886851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 848.904099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 851.424533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.428407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 851.454333] Lustre: Skipped 1 previous similar message [ 853.415220] LustreError: 35849:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 853.420528] LustreError: 35849:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 853.545820] Lustre: server umount lustre-MDT0000 complete [ 856.544477] LustreError: 33374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 856.551646] LustreError: 33374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 856.793622] LustreError: 33360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334419 with bad export cookie 6068771414902476051 [ 856.800632] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 856.807520] LustreError: 33360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 856.965532] LustreError: 36051:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 856.968406] LustreError: 36051:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 857.131979] Lustre: server umount lustre-MDT0001 complete [ 865.518035] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 867.532212] Key type lgssc unregistered [ 867.770235] LNet: 36591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.782648] LNetError: 36591:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.804285] LNet: Removed LNI 192.168.202.143@tcp [ 868.455548] Key type .llcrypt unregistered [ 868.457950] Key type ._llcrypt unregistered [ 886.799984] Key type ._llcrypt registered [ 886.801374] Key type .llcrypt registered [ 886.885676] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 887.523820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 887.607962] alg: No test for adler32 (adler32-zlib) [ 888.583331] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 888.727751] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 890.367678] Key type lgssc registered [ 891.125707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 896.006725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 896.020947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 897.308206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.407491] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 904.311389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 904.541238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 907.021926] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 910.151352] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 912.188644] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 916.601829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 916.862693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 921.212238] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 921.984200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 926.297752] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 937.441459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 937.452093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 937.459165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 938.922537] Lustre: lustre-OST0000: Not available for connect from 192.168.202.43@tcp (stopping) [ 939.493351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.754094] LustreError: 40209:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 941.831317] Lustre: server umount lustre-OST0000 complete [ 960.992860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 961.011911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 961.019943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 961.022817] Lustre: Skipped 2 previous similar messages [ 964.064940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.066225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 966.586501] LustreError: 40584:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 966.605332] LustreError: 40584:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 966.716711] Lustre: server umount lustre-MDT0000 complete [ 969.187745] LustreError: 37962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.193967] LustreError: 37962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 970.425656] LustreError: 37942:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334533 with bad export cookie 7294560946977865524 [ 970.428327] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 970.438285] LustreError: 37942:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 976.672598] LustreError: 40787:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 976.675604] LustreError: 40787:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 976.821795] Lustre: server umount lustre-MDT0001 complete [ 987.699246] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 990.553589] Key type lgssc unregistered [ 990.882214] LNet: 41320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.896733] LNetError: 41320:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.909887] LNet: Removed LNI 192.168.202.143@tcp [ 991.714317] Key type .llcrypt unregistered [ 991.725573] Key type ._llcrypt unregistered [ 1010.874904] Key type ._llcrypt registered [ 1010.877189] Key type .llcrypt registered [ 1011.002410] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 1012.073363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1012.103657] alg: No test for adler32 (adler32-zlib) [ 1013.154174] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 1013.314196] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 1015.007170] Key type lgssc registered [ 1016.023822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1022.437328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1022.466183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.964595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1027.013995] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1033.796793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1034.101820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1037.182031] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1041.710959] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1044.946639] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1050.418993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1050.492189] LustreError: 44291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1050.508072] LustreError: 44291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1050.735571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1055.309879] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1056.269190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1060.115691] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1066.927832] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.43@tcp (stopping) [ 1067.606103] LustreError: 44936:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1067.727177] Lustre: server umount lustre-MDT0000 complete [ 1069.540647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.541638] LustreError: 42686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1069.559279] Lustre: Skipped 1 previous similar message [ 1069.577337] LustreError: 42686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1070.764261] LustreError: 42673:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334633 with bad export cookie 7676960742387634935 [ 1070.765504] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1070.770286] LustreError: 42673:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1072.027235] LustreError: 42688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.061397] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.43@tcp (stopping) [ 1074.656450] LustreError: 43478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.660432] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.699503] Lustre: Skipped 1 previous similar message [ 1074.705601] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1077.148271] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.43@tcp (stopping) [ 1077.154669] Lustre: Skipped 1 previous similar message [ 1080.800641] LustreError: 42687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1080.807654] LustreError: 42687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1082.263638] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.43@tcp (stopping) [ 1082.275529] Lustre: Skipped 1 previous similar message [ 1085.408584] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1085.489874] LustreError: 45139:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1085.520431] LustreError: 45139:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1085.714431] Lustre: server umount lustre-MDT0001 complete [ 1099.982240] LustreError: 45365:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1099.989844] LustreError: 45365:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1100.135629] Lustre: server umount lustre-OST0000 complete [ 1114.441351] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 1116.887544] Key type lgssc unregistered [ 1117.120306] LNet: 46201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.126460] LNetError: 46201:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.185689] LNet: Removed LNI 192.168.202.143@tcp [ 1118.930081] Key type .llcrypt unregistered [ 1118.932581] Key type ._llcrypt unregistered [ 1136.838705] Key type ._llcrypt registered [ 1136.840888] Key type .llcrypt registered [ 1136.989899] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 1137.871740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1137.890351] alg: No test for adler32 (adler32-zlib) [ 1138.914027] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 1139.166927] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 1140.857721] Key type lgssc registered [ 1141.618902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1147.279698] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1147.293706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1162.783342] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1167.905856] LustreError: 47547:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.143@tcp: failed processing log, type 4: rc = -110 [ 1194.591230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.621588] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1202.567861] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1369.216506] LustreError: 48255:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1369.321280] Lustre: server umount lustre-OST0000 complete [ 1380.005464] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 1381.688080] Key type lgssc unregistered [ 1381.865050] LNet: 49090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.872499] LNetError: 49090:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.889425] LNet: Removed LNI 192.168.202.143@tcp [ 1382.410180] Key type .llcrypt unregistered [ 1382.412043] Key type ._llcrypt unregistered [ 1399.764584] Key type ._llcrypt registered [ 1399.769539] Key type .llcrypt registered [ 1399.943354] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 1401.452563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1401.580640] alg: No test for adler32 (adler32-zlib) [ 1402.692586] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 1403.044231] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 1404.735215] Key type lgssc registered [ 1405.771584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1413.782903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1413.809568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.298629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.683779] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1424.865176] LustreError: 50466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1424.873764] LustreError: 50466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1425.639393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1425.936485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1428.804716] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1433.088392] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1435.866781] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1440.363042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1440.589240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1444.490885] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1448.678257] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1450.577630] LustreError: 52706:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1450.733146] Lustre: server umount lustre-OST0000 complete [ 1456.610247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.617660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.617756] Lustre: Skipped 1 previous similar message [ 1459.756475] LustreError: 52906:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1459.758990] LustreError: 52906:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1459.841720] Lustre: server umount lustre-MDT0000 complete [ 1461.729438] LustreError: 51177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.744551] LustreError: 51177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1462.921989] LustreError: 50446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335026 with bad export cookie 9600968565692137438 [ 1462.935246] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1462.938429] LustreError: 50446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1463.049355] LustreError: 53108:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1463.052437] LustreError: 53108:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1463.298549] Lustre: server umount lustre-MDT0001 complete [ 1472.919661] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 1475.471110] Key type lgssc unregistered [ 1475.766711] LNet: 53641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.779249] LNetError: 53641:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.801632] LNet: Removed LNI 192.168.202.143@tcp [ 1476.532924] Key type .llcrypt unregistered [ 1476.535508] Key type ._llcrypt unregistered [ 1497.239293] Key type ._llcrypt registered [ 1497.242314] Key type .llcrypt registered [ 1497.346512] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 1498.049192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1498.056946] alg: No test for adler32 (adler32-zlib) [ 1499.192949] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 1499.516150] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 1501.207182] Key type lgssc registered [ 1502.520672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1509.587609] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1509.615870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1525.215417] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1530.335197] LustreError: 55006:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.143@tcp: failed processing log, type 4: rc = -110 [ 1557.023182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.179485] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1566.719892] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1572.132139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.482402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.544749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1575.645298] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1577.953476] LustreError: 55033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1577.977127] LustreError: 55033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1581.933033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1582.384137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1586.060939] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1591.064690] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1594.047907] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1596.899548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.911957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1596.918910] Lustre: Skipped 1 previous similar message [ 1601.533367] LustreError: 57284:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1601.676040] Lustre: server umount lustre-OST0000 complete [ 1622.497821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1622.503791] Lustre: Skipped 2 previous similar messages [ 1622.506070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1622.508669] Lustre: Skipped 1 previous similar message [ 1625.624130] LustreError: 57665:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1625.628067] LustreError: 57665:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1625.713711] Lustre: server umount lustre-MDT0000 complete [ 1627.616876] LustreError: 57405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1627.630539] LustreError: 57405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1628.257933] LustreError: 56659:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335191 with bad export cookie 13892741090980776373 [ 1628.259110] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1628.263015] LustreError: 56659:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1634.470996] LustreError: 57867:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1634.528807] LustreError: 57867:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1634.702890] Lustre: server umount lustre-MDT0001 complete [ 1643.538850] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 1646.313753] Key type lgssc unregistered [ 1646.578551] LNet: 58399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.585051] LNetError: 58399:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.606772] LNet: Removed LNI 192.168.202.143@tcp [ 1647.363533] Key type .llcrypt unregistered [ 1647.365147] Key type ._llcrypt unregistered [ 1666.225397] Key type ._llcrypt registered [ 1666.229049] Key type .llcrypt registered [ 1666.290392] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 1667.047049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1667.114617] alg: No test for adler32 (adler32-zlib) [ 1668.156644] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 1668.329980] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 1669.968934] Key type lgssc registered [ 1670.686724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1675.620364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1675.639643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.012767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1679.792725] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1686.159526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1686.416739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1689.087710] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1693.351528] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1695.951834] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1701.290306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1701.556430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.578642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1705.928180] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1709.807345] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1727.461791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1727.462880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.470047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1727.483201] Lustre: Skipped 1 previous similar message [ 1731.447146] LustreError: 62037:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1731.546889] Lustre: server umount lustre-OST0000 complete [ 1737.700044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1737.700823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.713286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1737.722247] Lustre: Skipped 2 previous similar messages [ 1740.752792] LustreError: 62240:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1740.757195] LustreError: 62240:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1740.861445] Lustre: server umount lustre-MDT0000 complete [ 1742.816892] LustreError: 59766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.830874] LustreError: 59766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1744.059547] LustreError: 61388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335307 with bad export cookie 16386385732406502956 [ 1744.067680] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1744.073831] LustreError: 61388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1744.175472] LustreError: 62442:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1744.179742] LustreError: 62442:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1744.304555] Lustre: server umount lustre-MDT0001 complete [ 1751.848260] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 1753.614225] Key type lgssc unregistered [ 1753.826117] LNet: 62972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.831807] LNetError: 62972:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.859737] LNet: Removed LNI 192.168.202.143@tcp [ 1755.390356] Key type .llcrypt unregistered [ 1755.392103] Key type ._llcrypt unregistered [ 1777.987363] Key type ._llcrypt registered [ 1777.991364] Key type .llcrypt registered [ 1778.072913] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 1778.764116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1778.834479] alg: No test for adler32 (adler32-zlib) [ 1779.817422] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 1780.015670] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 1781.679198] Key type lgssc registered [ 1782.694963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1788.771141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1788.791499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.266406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.556385] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1800.430510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1800.727601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1803.701740] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1807.584363] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1810.441385] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1814.829697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1815.087216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.110285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1819.191952] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1823.036883] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1831.488248] Lustre: Failing over lustre-MDT0000 [ 1831.617240] LustreError: 66881:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1831.715350] Lustre: server umount lustre-MDT0000 complete [ 1834.941429] LustreError: 65846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1834.947574] LustreError: 65846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1835.490408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.497733] LustreError: 64637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1835.532826] Lustre: Skipped 1 previous similar message [ 1839.489512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1839.577568] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1839.696062] Lustre: *** cfs_fail_loc=135, val=0*** [ 1839.819430] LustreError: 67391:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1839.824623] LustreError: 67391:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.143@tcp: cfg command failed: rc = -2 [ 1839.831544] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1839.831544] [ 1839.847059] LustreError: MGC192.168.202.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1839.857175] LustreError: 67379:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1839.866986] LustreError: 67379:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1839.870567] LustreError: 67379:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1839.891765] Lustre: server umount lustre-MDT0000 complete [ 1839.894587] LustreError: 67379:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1840.032778] LustreError: 65846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1840.055507] LustreError: 65846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1844.709397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.819613] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1845.076332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.163293] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1848.019316] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1850.262406] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1850.341929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1850.346525] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1850.381155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1855.456652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1855.461779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.469441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1855.476465] Lustre: Skipped 2 previous similar messages [ 1855.484039] Lustre: Skipped 1 previous similar message [ 1859.337117] LustreError: 68201:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1859.340270] LustreError: 68201:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1859.496926] Lustre: server umount lustre-OST0000 complete [ 1865.697270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1865.698390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.703703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1865.712552] Lustre: Skipped 1 previous similar message [ 1868.272632] LustreError: 68402:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1868.278311] LustreError: 68402:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1868.313309] Lustre: server umount lustre-MDT0000 complete [ 1870.818467] LustreError: 64636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1870.827534] LustreError: 64636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1871.465498] LustreError: 64616:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335434 with bad export cookie 9746805278427013860 [ 1871.472020] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.475742] LustreError: 64616:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1871.593217] LustreError: 68605:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1871.602400] LustreError: 68605:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1871.769519] Lustre: server umount lustre-MDT0001 complete [ 1880.171210] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 1882.646428] Key type lgssc unregistered [ 1882.915181] LNet: 69135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.927267] LNetError: 69135:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.956799] LNet: Removed LNI 192.168.202.143@tcp [ 1883.767816] Key type .llcrypt unregistered [ 1883.770069] Key type ._llcrypt unregistered [ 1902.150224] Key type ._llcrypt registered [ 1902.152077] Key type .llcrypt registered [ 1902.299419] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 1903.251289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.273253] alg: No test for adler32 (adler32-zlib) [ 1904.288568] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 1904.509703] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 1906.191394] Key type lgssc registered [ 1907.452763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1914.004951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1914.019440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1915.343508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.917247] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1925.600740] LustreError: 70507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1925.613718] LustreError: 70507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1926.034930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1926.358194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1930.013614] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1934.912773] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1938.348297] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1943.876945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1944.177127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.739435] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1953.412748] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1960.566632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1969.041685] Lustre: Failing over lustre-MDT0000 [ 1969.210792] LustreError: 72774:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1969.266511] Lustre: server umount lustre-MDT0000 complete [ 1970.618870] LustreError: 70502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.170037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.178781] Lustre: Skipped 2 previous similar messages [ 1975.703675] LustreError: 70502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1975.720491] LustreError: 70502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1977.029167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.194705] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1977.352851] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1977.525444] LustreError: 73283:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1977.529251] LustreError: 73283:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.143@tcp: cfg command failed: rc = -9 [ 1977.537271] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1977.537271] [ 1977.544940] LustreError: MGC192.168.202.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1977.556268] LustreError: 73271:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1977.559995] LustreError: 73271:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1977.566372] LustreError: 73271:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1977.583641] Lustre: server umount lustre-MDT0000 complete [ 1977.588309] LustreError: 73271:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1980.824189] LustreError: 70503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.847636] LustreError: 70503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1983.705569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.925391] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1984.175909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.227723] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1985.945858] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1987.572701] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1989.613788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1989.646104] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1989.706513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1994.727658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.765601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1994.778724] Lustre: Skipped 1 previous similar message [ 1996.487953] LustreError: 74096:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1996.490157] LustreError: 74096:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1996.642077] Lustre: server umount lustre-OST0000 complete [ 2004.962422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.965080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2004.980966] Lustre: Skipped 1 previous similar message [ 2004.991542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.080135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.085613] Lustre: Skipped 2 previous similar messages [ 2015.199243] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2015.203931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.232481] Lustre: Skipped 1 previous similar message [ 2015.308510] LustreError: 74298:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2015.317657] LustreError: 74298:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2015.410020] Lustre: server umount lustre-MDT0000 complete [ 2019.015129] LustreError: 72775:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335582 with bad export cookie 3919044714066653134 [ 2019.019372] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.027908] LustreError: 72775:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2019.284718] LustreError: 74500:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2019.287862] LustreError: 74500:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2019.529642] Lustre: server umount lustre-MDT0001 complete [ 2027.339874] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 2029.062510] Key type lgssc unregistered [ 2029.266491] LNet: 75033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.271469] LNetError: 75033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2029.281821] LNet: Removed LNI 192.168.202.143@tcp [ 2029.811353] Key type .llcrypt unregistered [ 2029.813855] Key type ._llcrypt unregistered [ 2039.251572] Key type ._llcrypt registered [ 2039.253601] Key type .llcrypt registered [ 2039.333486] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 2039.530461] Key type .llcrypt unregistered [ 2039.532062] Key type ._llcrypt unregistered [ 2055.749762] Key type ._llcrypt registered [ 2055.751135] Key type .llcrypt registered [ 2055.840485] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2056.366820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.403274] alg: No test for adler32 (adler32-zlib) [ 2057.429948] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 2057.613384] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 2059.287190] Key type lgssc registered [ 2060.091963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2065.997622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2066.013658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.343465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.030990] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2076.740935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2077.246979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2080.422864] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2085.569358] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2089.486560] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2095.264312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2095.569228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.123891] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2104.588138] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2113.016082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2123.234723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2123.235611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.238041] LustreError: Skipped 1 previous similar message [ 2123.246099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2123.253829] Lustre: Skipped 1 previous similar message [ 2127.300227] LustreError: 79143:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2127.423500] Lustre: server umount lustre-OST0000 complete [ 2133.473797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2133.477377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.498657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2133.511197] Lustre: Skipped 2 previous similar messages [ 2136.561269] LustreError: 79345:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2136.585174] LustreError: 79345:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2136.665938] Lustre: server umount lustre-MDT0000 complete [ 2138.592521] LustreError: 76878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.616238] LustreError: 76878:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2139.965679] LustreError: 76857:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335703 with bad export cookie 6117965659026297539 [ 2139.981439] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.992097] LustreError: 76857:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2140.107925] LustreError: 79546:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2140.122702] LustreError: 79546:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2140.324374] Lustre: server umount lustre-MDT0001 complete [ 2149.197847] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 2151.699267] Key type lgssc unregistered [ 2152.011939] LNet: 80080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.015932] LNetError: 80080:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2152.027661] LNet: Removed LNI 192.168.202.143@tcp [ 2152.810683] Key type .llcrypt unregistered [ 2152.813121] Key type ._llcrypt unregistered [ 2170.719954] Key type ._llcrypt registered [ 2170.723937] Key type .llcrypt registered [ 2170.833711] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2171.607664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2171.621345] alg: No test for adler32 (adler32-zlib) [ 2172.640925] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 2172.869525] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 2174.495165] Key type lgssc registered [ 2175.491754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2183.015282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2183.048579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2184.453868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.896912] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2193.894553] LustreError: 81453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.916313] LustreError: 81453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2196.064304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2196.512040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2199.441199] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2205.103649] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2209.036376] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2217.290809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2217.615442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.955953] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2228.260954] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2229.899853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2235.362565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.367327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2235.413767] Lustre: Skipped 1 previous similar message [ 2235.434378] Lustre: Skipped 1 previous similar message [ 2237.058897] LustreError: 83699:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2237.168796] Lustre: server umount lustre-OST0000 complete [ 2240.484203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.489917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2240.499394] Lustre: Skipped 1 previous similar message [ 2245.601827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2245.613433] Lustre: Skipped 2 previous similar messages [ 2246.374410] LustreError: 83901:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2246.384348] LustreError: 83901:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2246.522258] Lustre: server umount lustre-MDT0000 complete [ 2249.522436] LustreError: 81435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335812 with bad export cookie 3352581032223565998 [ 2249.528828] LustreError: 81435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2249.531679] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2249.628190] LustreError: 84105:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2249.630047] LustreError: 84105:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2249.725861] Lustre: server umount lustre-MDT0001 complete [ 2257.888912] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 2260.034129] Key type lgssc unregistered [ 2260.277274] LNet: 84637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2260.282931] LNetError: 84637:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2260.294587] LNet: Removed LNI 192.168.202.143@tcp [ 2260.947174] Key type .llcrypt unregistered [ 2260.948647] Key type ._llcrypt unregistered [ 2279.235180] Key type ._llcrypt registered [ 2279.236930] Key type .llcrypt registered [ 2279.312602] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2279.998368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.006445] alg: No test for adler32 (adler32-zlib) [ 2280.941528] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 2281.071176] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 2282.703177] Key type lgssc registered [ 2283.566775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2289.458533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2289.480741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.838158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2293.874145] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2296.294395] LustreError: 86011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.320088] LustreError: 86011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2300.452503] LustreError: 86011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.745021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2301.087797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2304.178141] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2308.450890] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2311.085543] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2316.735368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2317.054551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2321.522750] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2325.100871] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2335.472417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2345.958439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.962118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2345.974239] Lustre: Skipped 1 previous similar message [ 2345.975804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2348.481483] LustreError: 88280:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2348.595496] Lustre: server umount lustre-OST0000 complete [ 2351.562545] LustreError: 88481:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2351.566331] LustreError: 88481:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2351.616518] Lustre: server umount lustre-MDT0000 complete [ 2352.104054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2352.107096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.133363] LustreError: 86011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.967059] LustreError: 85990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335917 with bad export cookie 18386874501779463125 [ 2353.975839] LustreError: 85990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2353.977682] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2360.152108] LustreError: 88682:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2360.165152] LustreError: 88682:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2360.317285] Lustre: server umount lustre-MDT0001 complete [ 2366.986657] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 2368.813882] Key type lgssc unregistered [ 2369.037925] LNet: 89214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.042751] LNetError: 89214:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.060332] LNet: Removed LNI 192.168.202.143@tcp [ 2369.688542] Key type .llcrypt unregistered [ 2369.690184] Key type ._llcrypt unregistered [ 2385.514997] Key type ._llcrypt registered [ 2385.516500] Key type .llcrypt registered [ 2385.584576] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2386.491934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2386.524485] alg: No test for adler32 (adler32-zlib) [ 2387.617240] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 2387.884207] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 2389.527760] Key type lgssc registered [ 2390.715384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2397.484958] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2397.499941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2412.959400] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2418.079190] LustreError: 90559:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.143@tcp: failed processing log, type 4: rc = -110 [ 2444.703413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.898951] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2452.567150] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2456.402279] LustreError: 91415:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2456.455975] Lustre: server umount lustre-OST0000 complete [ 2462.297868] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:33:45 (1763336025) [ 2467.435796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2467.856726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.917514] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2472.927963] LustreError: 92087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.942263] LustreError: 92087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2477.026281] LustreError: 92087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2478.098961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2478.368192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2481.180881] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2485.231962] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2488.313465] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2494.084597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2494.343883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.511539] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2503.155658] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2512.821576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2523.113920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2523.114755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2523.131238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2523.149242] Lustre: Skipped 1 previous similar message [ 2528.228351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2528.230484] Lustre: Skipped 1 previous similar message [ 2528.585458] LustreError: 94513:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2528.594218] LustreError: 94513:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2528.689974] Lustre: server umount lustre-OST0000 complete [ 2533.344444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2533.355331] Lustre: Skipped 1 previous similar message [ 2533.367047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2533.372471] Lustre: Skipped 1 previous similar message [ 2538.479362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2538.487334] Lustre: Skipped 1 previous similar message [ 2538.517851] LustreError: 94715:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2538.526049] LustreError: 94715:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2538.608222] Lustre: server umount lustre-MDT0000 complete [ 2541.420971] LustreError: 94427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336104 with bad export cookie 16832189421225907470 [ 2541.428502] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2541.547992] LustreError: 94917:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2541.551801] LustreError: 94917:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2541.654272] Lustre: server umount lustre-MDT0001 complete [ 2550.163346] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 2552.366236] Key type lgssc unregistered [ 2552.601410] LNet: 95449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2552.610289] LNetError: 95449:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2552.620546] LNet: Removed LNI 192.168.202.143@tcp [ 2553.107512] Key type .llcrypt unregistered [ 2553.109426] Key type ._llcrypt unregistered [ 2569.273993] Key type ._llcrypt registered [ 2569.275137] Key type .llcrypt registered [ 2569.368207] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2570.273856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2570.288915] alg: No test for adler32 (adler32-zlib) [ 2571.251214] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 2571.422303] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 2573.063150] Key type lgssc registered [ 2573.943465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2580.636670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2580.670231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.025595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.204440] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2592.235495] LustreError: 96822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.266844] LustreError: 96822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2593.082324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2593.492508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2596.551796] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2600.968759] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2604.101733] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2609.574455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2609.881276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2614.643682] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2619.877197] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2627.315929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2638.816711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2638.820075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2638.838941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2642.918253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2642.919921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2642.922627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2642.929291] Lustre: Skipped 1 previous similar message [ 2644.360283] LustreError: 99088:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2644.492394] Lustre: server umount lustre-OST0000 complete [ 2650.080341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2650.083857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.089729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2653.154277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2653.158982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.273727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2658.279051] Lustre: Skipped 2 previous similar messages [ 2662.367167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2662.468598] LustreError: 99291:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2662.473482] LustreError: 99291:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2662.566824] Lustre: server umount lustre-MDT0000 complete [ 2663.392220] LustreError: 96823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.411880] LustreError: 96823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2665.885983] LustreError: 96803:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336229 with bad export cookie 12259435207589974418 [ 2665.887025] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2665.907492] LustreError: 96803:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2666.100181] LustreError: 99493:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2666.111994] LustreError: 99493:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2666.371211] Lustre: server umount lustre-MDT0001 complete [ 2675.269117] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 2678.091190] Key type lgssc unregistered [ 2678.369787] LNet: 100025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2678.387647] LNetError: 100025:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2678.411710] LNet: Removed LNI 192.168.202.143@tcp [ 2679.090405] Key type .llcrypt unregistered [ 2679.092085] Key type ._llcrypt unregistered [ 2696.063450] Key type ._llcrypt registered [ 2696.065067] Key type .llcrypt registered [ 2696.144455] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2697.075790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2697.109351] alg: No test for adler32 (adler32-zlib) [ 2698.254299] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 2698.566351] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 2700.263191] Key type lgssc registered [ 2701.301710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2708.800894] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2708.817206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2724.447371] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2729.567202] LustreError: 101247:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.143@tcp: failed processing log, type 4: rc = -110 [ 2756.191216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.953674] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2765.644559] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2771.951113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2772.302302] LustreError: MGC192.168.202.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2772.309168] LustreError: 102159:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2772.327750] LustreError: 102159:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2772.336791] LustreError: 102159:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2772.414384] Lustre: server umount lustre-MDT0000 complete [ 2772.416551] LustreError: 102159:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2792.927153] Lustre: 100588:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763336340/real 1763336340] req@00000000d2508e17 x1848992083217536/t0(0) o400->MGC192.168.202.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763336356 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2792.927840] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2792.952037] Lustre: 100588:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2792.964432] LustreError: 101261:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2792.994337] LustreError: 102591:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2793.134590] Lustre: server umount lustre-OST0000 complete [ 2800.331379] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 2808.172947] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2816.490333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2821.845566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2826.748046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2831.766881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2837.882731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2838.009537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.323336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2838.357523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2838.455430] Lustre: lustre-MDT0000: new disk, initializing [ 2838.554636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.579376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2842.192662] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2851.004967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2851.078834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2851.155402] Lustre: 104931:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2851.207902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2851.210732] Lustre: Skipped 1 previous similar message [ 2851.258958] Lustre: lustre-MDT0001: new disk, initializing [ 2851.328651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2851.380560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2851.391421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2855.044138] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2861.239656] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2864.397951] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2870.497495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2870.570882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2870.799038] Lustre: lustre-OST0000: new disk, initializing [ 2870.816602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2870.897695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2875.790173] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2880.055218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2880.071679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2880.179105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2882.662287] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2886.504590] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2886.714829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2890.569886] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2890.756292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2895.844220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2895.846276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.848133] LustreError: Skipped 1 previous similar message [ 2895.849046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2895.869908] Lustre: Skipped 1 previous similar message [ 2898.761233] LustreError: 107870:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2898.773340] LustreError: 107870:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2898.957432] Lustre: server umount lustre-OST0000 complete [ 2906.082824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2906.089496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.098112] Lustre: Skipped 1 previous similar message [ 2906.103325] Lustre: Skipped 1 previous similar message [ 2911.210801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2911.213729] Lustre: Skipped 2 previous similar messages [ 2916.329566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2917.343152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2917.458736] LustreError: 108071:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2917.472435] LustreError: 108071:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2917.541993] Lustre: server umount lustre-MDT0000 complete [ 2920.887660] LustreError: 104923:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336484 with bad export cookie 6688292298815354399 [ 2920.893058] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2920.902542] LustreError: 104923:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2921.060621] LustreError: 108274:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2921.063343] LustreError: 108274:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2921.232962] Lustre: server umount lustre-MDT0001 complete [ 2928.699070] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:41:32 (1763336492) [ 2931.332809] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2944.277052] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 2952.083687] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2959.184394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2963.600727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2968.473312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2973.565313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2980.101250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2980.146623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2980.311547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2980.343932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2980.430838] Lustre: lustre-MDT0000: new disk, initializing [ 2980.542900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2980.574237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2984.039983] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2992.602771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2992.699493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2992.847812] Lustre: 111408:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2992.949141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2992.960989] Lustre: Skipped 1 previous similar message [ 2993.042505] Lustre: lustre-MDT0001: new disk, initializing [ 2993.119407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2993.161969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2993.173627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2996.496960] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3002.953612] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3005.759530] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3012.011260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3012.115420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3012.450515] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3012.458687] Lustre: Skipped 1 previous similar message [ 3012.464125] Lustre: lustre-OST0000: new disk, initializing [ 3012.476523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3012.595189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3017.671529] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3017.723671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3017.735729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3017.808659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3024.698489] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3028.579193] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3028.768762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3032.117703] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3032.295046] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3038.181247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.188638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3038.199444] Lustre: Skipped 1 previous similar message [ 3038.208139] Lustre: Skipped 1 previous similar message [ 3040.062274] LustreError: 114356:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3040.065535] LustreError: 114356:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3040.213934] Lustre: server umount lustre-OST0000 complete [ 3044.320428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3044.333342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3048.417852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3050.164824] LustreError: 114558:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 3050.167894] LustreError: 114558:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3050.303949] Lustre: server umount lustre-MDT0000 complete [ 3053.707824] LustreError: 111400:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336616 with bad export cookie 6688292298815355281 [ 3053.715818] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3054.089312] Lustre: server umount lustre-MDT0001 complete [ 3062.070146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.417051] LustreError: 115238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.436396] LustreError: 115238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3062.492730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.774042] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3067.876129] LustreError: 115238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.969088] LustreError: 115238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.919683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3078.854160] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3085.220822] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3088.417158] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3094.205845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3099.265462] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3104.678984] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3114.661598] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3118.050861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3118.053974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3118.069664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3118.073033] Lustre: Skipped 3 previous similar messages [ 3123.277117] LustreError: 117579:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3123.279637] LustreError: 117579:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3123.352537] Lustre: server umount lustre-OST0000 complete [ 3126.757537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3132.320326] Lustre: server umount lustre-MDT0000 complete [ 3133.408859] LustreError: 115238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.430207] LustreError: 115238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3135.272951] LustreError: 115218:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336698 with bad export cookie 6688292298815356163 [ 3135.278720] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3135.283917] LustreError: 115218:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3135.496994] Lustre: server umount lustre-MDT0001 complete [ 3145.924318] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 3148.290822] Key type lgssc unregistered [ 3148.497302] LNet: 118518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3148.502338] LNetError: 118518:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3148.513613] LNet: Removed LNI 192.168.202.143@tcp [ 3149.230449] Key type .llcrypt unregistered [ 3149.231721] Key type ._llcrypt unregistered [ 3163.291564] Key type ._llcrypt registered [ 3163.293545] Key type .llcrypt registered [ 3163.412889] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 3174.319910] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 3174.802450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3174.871487] alg: No test for adler32 (adler32-zlib) [ 3175.848088] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 3176.047536] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 3177.687181] Key type lgssc registered [ 3178.610759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3186.646332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3193.194185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3199.841788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3206.080366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3212.736828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3212.804846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3212.829859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3214.047632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3214.088242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3214.157972] Lustre: lustre-MDT0000: new disk, initializing [ 3214.223130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3214.232430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3217.348302] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3225.834642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3225.908785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3226.025189] Lustre: 121622:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3226.068997] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3226.072266] Lustre: Skipped 1 previous similar message [ 3226.161882] Lustre: lustre-MDT0001: new disk, initializing [ 3226.255068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3226.295868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3226.304862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3229.308057] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3234.884690] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3237.352483] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3242.345378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3242.392703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3242.556469] Lustre: lustre-OST0000: new disk, initializing [ 3242.565192] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3242.636166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3246.393986] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3247.652896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3247.660245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3247.728746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3251.843834] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3255.377660] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3255.602066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3258.916752] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3259.129856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3262.947150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3262.953048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3262.960317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3262.967442] Lustre: Skipped 1 previous similar message [ 3268.070206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3268.072711] Lustre: Skipped 1 previous similar message [ 3273.184986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3273.193551] Lustre: Skipped 1 previous similar message [ 3275.231160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3275.299867] LustreError: 124562:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3275.398290] Lustre: server umount lustre-OST0000 complete [ 3282.916578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3282.926314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.945895] Lustre: Skipped 1 previous similar message [ 3282.952901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3284.598024] LustreError: 124765:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3284.602906] LustreError: 124765:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3284.694076] Lustre: server umount lustre-MDT0000 complete [ 3287.658664] LustreError: 121613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336850 with bad export cookie 10066716885418121487 [ 3287.659877] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3287.663852] LustreError: 121613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3287.778782] LustreError: 124967:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3287.791357] LustreError: 124967:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3287.988972] Lustre: server umount lustre-MDT0001 complete [ 3294.121872] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:47:37 (1763336857) [ 3299.738528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3300.159284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3303.108495] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3305.443426] LustreError: 125637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3305.466229] LustreError: 125637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3309.285831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3309.552707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3312.464918] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3316.729189] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3319.563475] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3324.896870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3324.897370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.903094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3324.906281] Lustre: Skipped 2 previous similar messages [ 3324.909018] Lustre: Skipped 2 previous similar messages [ 3327.205033] LustreError: 126981:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3327.207866] LustreError: 126981:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3327.279842] Lustre: server umount lustre-MDT0000 complete [ 3330.017139] LustreError: 126356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3330.033385] LustreError: 126356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3330.367331] LustreError: 125618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336893 with bad export cookie 10066716885418122362 [ 3330.379184] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3330.392311] LustreError: 125618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3330.660326] Lustre: server umount lustre-MDT0001 complete [ 3339.111977] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:48:22 (1763336902) [ 3347.185298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3362.788846] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3367.904071] LustreError: 127829:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.143@tcp: failed processing log, type 4: rc = -110 [ 3393.631329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3398.255557] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3403.229040] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3405.186818] LustreError: 128493:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3405.189884] LustreError: 128493:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3405.340206] Lustre: server umount lustre-OST0000 complete [ 3413.108149] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:49:36 (1763336976) [ 3418.568179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3418.945955] LustreError: 129166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3419.005636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3422.157733] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3424.224221] LustreError: 129166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.348502] LustreError: 129166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.735420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3432.770082] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3436.844284] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3439.701282] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3445.583860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3450.546451] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3454.930153] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3469.794228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.797522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3469.802788] Lustre: Skipped 2 previous similar messages [ 3469.815863] Lustre: Skipped 3 previous similar messages [ 3475.009081] LustreError: 131432:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3475.020406] LustreError: 131432:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3475.074104] Lustre: server umount lustre-MDT0000 complete [ 3477.690521] LustreError: 129147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337040 with bad export cookie 10066716885418123034 [ 3477.691471] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3477.695710] LustreError: 129147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3477.945546] Lustre: server umount lustre-MDT0001 complete [ 3490.174690] Lustre: server umount lustre-OST0000 complete [ 3497.488834] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:51:00 (1763337060) [ 3502.721548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.046949] LustreError: 132508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3503.111336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.115378] Lustre: Skipped 2 previous similar messages [ 3505.956850] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3512.484653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3515.533759] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3519.655436] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3522.832358] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3528.882072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3534.297955] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3538.663331] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3541.982482] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3548.716415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3549.545734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3552.653552] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3552.821502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3557.860651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3557.872129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3557.893721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3557.896357] Lustre: Skipped 3 previous similar messages [ 3560.452379] LustreError: 135299:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3560.461917] LustreError: 135299:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3560.585364] Lustre: server umount lustre-OST0000 complete [ 3564.004794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3564.012239] LustreError: Skipped 1 previous similar message [ 3578.335181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3578.457996] Lustre: server umount lustre-MDT0000 complete [ 3579.363034] LustreError: 132507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3579.378681] LustreError: 132507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3581.351715] LustreError: 133226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337144 with bad export cookie 10066716885418124105 [ 3581.356057] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3581.359817] LustreError: 133226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3581.698851] Lustre: server umount lustre-MDT0001 complete [ 3587.521825] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:52:31 (1763337151) [ 3592.542832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3608.031702] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3613.152834] LustreError: 136348:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.143@tcp: failed processing log, type 4: rc = -110 [ 3638.879321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3638.886090] Lustre: Skipped 2 previous similar messages [ 3643.867543] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3649.072296] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3654.787675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.089742] LustreError: 136375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3655.231047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3658.699632] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3666.350902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3670.274181] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3675.363801] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3678.328822] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3681.903815] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3682.098371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3685.365823] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3685.539564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3687.393586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3687.397442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.403708] Lustre: Skipped 3 previous similar messages [ 3687.413775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3687.421245] Lustre: Skipped 8 previous similar messages [ 3693.357235] LustreError: 139122:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3693.363332] LustreError: 139122:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3693.462369] Lustre: server umount lustre-MDT0000 complete [ 3696.371839] LustreError: 136355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337259 with bad export cookie 10066716885418125127 [ 3696.389980] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3696.573277] Lustre: server umount lustre-MDT0001 complete [ 3709.792020] Lustre: server umount lustre-OST0000 complete [ 3716.880086] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:54:40 (1763337280) [ 3722.777439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3738.271737] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3743.391548] LustreError: 140171:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.143@tcp: failed processing log, type 4: rc = -110 [ 3768.991286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3768.994351] Lustre: Skipped 2 previous similar messages [ 3773.771266] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3778.357416] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3783.198498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.498381] LustreError: 140198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3783.510365] LustreError: 140198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3783.665279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3786.814800] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3794.571505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3799.392055] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3804.809136] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3808.349543] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3812.989812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3813.044524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3813.142623] Lustre: lustre-OST0001: new disk, initializing [ 3813.150406] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3817.724904] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3818.536896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3818.542405] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3818.613699] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3823.838462] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3827.443330] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3827.597258] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3830.782104] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3830.979163] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3833.831096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.858115] Lustre: Skipped 2 previous similar messages [ 3833.870745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3833.873586] Lustre: Skipped 4 previous similar messages [ 3838.794191] LustreError: 143901:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3838.799732] LustreError: 143901:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3838.849099] Lustre: server umount lustre-OST0000 complete [ 3844.065445] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3844.073201] LustreError: Skipped 1 previous similar message [ 3856.351140] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3856.588826] Lustre: server umount lustre-OST0001 complete [ 3865.989521] Lustre: server umount lustre-MDT0000 complete [ 3869.196576] LustreError: 142704:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337432 with bad export cookie 10066716885418126072 [ 3869.200179] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3869.204669] LustreError: 142704:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3869.485784] Lustre: server umount lustre-MDT0001 complete [ 3874.538209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3877.964930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3880.931436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3884.830410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3891.999050] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:57:34 (1763337454) [ 3893.651140] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3895.331841] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3897.276869] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:57:40 (1763337460) [ 3903.457427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3903.804792] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3903.856996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3908.309650] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3916.804422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3916.913318] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3916.945361] Lustre: 146144:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3921.077983] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3925.793625] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3929.107659] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3935.871783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3936.122477] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3941.593555] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3943.033523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3946.620308] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3950.702208] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3950.914874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3954.340315] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3954.528326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3956.203027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3956.218338] Lustre: Skipped 5 previous similar messages [ 3962.442975] Lustre: server umount lustre-OST0000 complete [ 3982.866580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3986.279333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3986.309656] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3986.311132] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3988.269479] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3993.555629] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3998.161642] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3998.340626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4002.013051] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4002.238645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4006.347854] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99f7e0e68000.ost_server_uuid 50 [ 4008.033579] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff99f7e0e68000.ost_server_uuid in FULL state after 0 sec [ 4008.218371] LustreError: 148081:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.202.43@tcp) returned error from blocking AST (req@00000000228abbb1 x1848992584670848 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f17e6f61/0x8bb429abe9c6344b lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.43@tcp remote: 0x61ca55d3811b1d02 expref: 5 pid: 146153 timeout: 4108 lvb_type: 0 [ 4008.242059] LustreError: lustre-MDT0000: A client on nid 192.168.202.43@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4008.250921] LustreError: 146141:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.43@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f17e6f61/0x8bb429abe9c6344b lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.43@tcp remote: 0x61ca55d3811b1d02 expref: 6 pid: 146153 timeout: 0 lvb_type: 0 [ 4012.003122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4012.010487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4012.016900] LustreError: Skipped 1 previous similar message [ 4012.032647] Lustre: Skipped 14 previous similar messages [ 4017.598124] LustreError: 150696:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4017.603278] LustreError: 150696:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 4017.785634] Lustre: server umount lustre-OST0000 complete [ 4022.242109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4022.252876] Lustre: Skipped 4 previous similar messages [ 4027.470480] Lustre: server umount lustre-MDT0000 complete [ 4031.179333] LustreError: 146872:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337594 with bad export cookie 10066716885418127248 [ 4031.191364] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4031.206969] LustreError: 146872:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4037.539893] Lustre: server umount lustre-MDT0001 complete [ 4047.155230] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 4049.831760] Key type lgssc unregistered [ 4050.136853] LNet: 151633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4050.144946] LNetError: 151633:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4050.171641] LNet: Removed LNI 192.168.202.143@tcp [ 4050.864641] Key type .llcrypt unregistered [ 4050.867981] Key type ._llcrypt unregistered [ 4071.040769] Key type ._llcrypt registered [ 4071.043081] Key type .llcrypt registered [ 4071.190890] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 4072.087365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4072.143305] alg: No test for adler32 (adler32-zlib) [ 4073.188777] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 4073.446779] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 4075.088350] Key type lgssc registered [ 4075.935554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4082.013013] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4082.033847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4083.432160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4087.314686] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4092.897874] LustreError: 153008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4092.908250] LustreError: 153008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4096.083632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4096.557242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4100.571864] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4105.820962] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4109.620818] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4116.341619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4116.558853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4122.570501] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4128.936823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4129.032192] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4136.301175] Lustre: Failing over lustre-MDT0000 [ 4136.471322] LustreError: 155267:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4136.568827] Lustre: server umount lustre-MDT0000 complete [ 4136.895815] LustreError: 155129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.442524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4137.453037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4139.500235] LustreError: 153007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4139.502444] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4139.507434] LustreError: 153007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4139.520175] Lustre: Skipped 1 previous similar message [ 4144.609854] LustreError: 154228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4144.643934] LustreError: 154228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4148.767992] LustreError: 152989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.43@tcp arrived at 1763337711 with bad export cookie 6472933582604703589 [ 4149.738808] LustreError: 153720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4149.781366] LustreError: 153720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4155.873642] Lustre: 152318:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763337702/real 1763337702] req@00000000c808a75f x1848993525031296/t0(0) o400->MGC192.168.202.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763337718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4155.915829] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4156.151341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.994452] LustreError: 154228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.011145] LustreError: 154228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4166.116324] LustreError: 152316:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009c7e376e x1848993525037312/t0(0) o250->MGC192.168.202.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4166.586957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4166.657301] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4170.741786] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4171.752759] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4171.762171] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4172.827415] Lustre: lustre-MDT0000: Denying connection for new client 22a7b9bf-f09a-442b-a615-21ebe5468d60 (at 192.168.202.43@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4181.986188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.995847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4182.013263] Lustre: Skipped 1 previous similar message [ 4186.761190] LustreError: 156207:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4186.770877] LustreError: 156207:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4186.917645] Lustre: server umount lustre-OST0000 complete [ 4190.612086] LustreError: 156408:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4190.615829] Lustre: 155774:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4190.622781] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4190.640234] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4190.647572] LustreError: 155774:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004d7f5546 x1848993525050880/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4190.655166] LustreError: 155774:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4190.660427] LustreError: 155774:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4190.665220] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 4190.810780] LustreError: 156408:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4190.821839] LustreError: 156408:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4190.899966] Lustre: server umount lustre-MDT0000 complete [ 4192.230463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.242572] LustreError: 153003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.271851] LustreError: 153003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4194.468030] LustreError: 152988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337757 with bad export cookie 6472933582604703687 [ 4194.469581] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4194.479734] LustreError: 152988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4196.634744] LustreError: 156610:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4196.641847] LustreError: 156610:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4196.871714] Lustre: server umount lustre-MDT0001 complete [ 4204.846098] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 4207.237586] Key type lgssc unregistered [ 4207.510791] LNet: 157141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4207.514887] LNetError: 157141:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4207.538714] LNet: Removed LNI 192.168.202.143@tcp [ 4208.242752] Key type .llcrypt unregistered [ 4208.248250] Key type ._llcrypt unregistered [ 4230.087777] Key type ._llcrypt registered [ 4230.089923] Key type .llcrypt registered [ 4230.186939] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 4231.399753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4231.461676] alg: No test for adler32 (adler32-zlib) [ 4232.485660] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 4232.684527] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 4234.319134] Key type lgssc registered [ 4235.308794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4242.068480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4242.081597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4243.497132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4246.819183] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4253.666295] LustreError: 158533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4253.682974] LustreError: 158533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4254.251704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4254.553792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4257.628224] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4262.175863] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4265.459934] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4271.217570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4271.492602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.781173] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4280.704492] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4288.889493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4299.238179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4299.239171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.242163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4299.260277] Lustre: Skipped 1 previous similar message [ 4302.023183] LustreError: 160804:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4302.174594] Lustre: server umount lustre-OST0000 complete [ 4305.888097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4305.901060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4305.935730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4305.938217] Lustre: Skipped 1 previous similar message [ 4309.472338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4309.477613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4311.291874] LustreError: 161005:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4311.311371] LustreError: 161005:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4311.387167] Lustre: server umount lustre-MDT0000 complete [ 4314.592764] LustreError: 159245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4314.608609] LustreError: 159245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4315.390051] LustreError: 158512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337878 with bad export cookie 9399987337512237234 [ 4315.393565] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4315.397187] LustreError: 158512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4315.553470] LustreError: 161206:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4315.557152] LustreError: 161206:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4315.767378] Lustre: server umount lustre-MDT0001 complete [ 4326.531053] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 4329.377607] Key type lgssc unregistered [ 4329.623173] LNet: 161739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4329.629356] LNetError: 161739:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4329.645852] LNet: Removed LNI 192.168.202.143@tcp [ 4330.507789] Key type .llcrypt unregistered [ 4330.509830] Key type ._llcrypt unregistered [ 4364.473355] Key type ._llcrypt registered [ 4364.474900] Key type .llcrypt registered [ 4364.661124] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 4365.770873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4365.790456] alg: No test for adler32 (adler32-zlib) [ 4367.015064] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 4367.285221] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 4368.991260] Key type lgssc registered [ 4370.068925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4377.870277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4377.903195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4379.455021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4383.951428] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4389.858184] LustreError: 163423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.881344] LustreError: 163423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4392.556889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4393.100842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4397.660539] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4404.235341] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4408.072910] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4414.940777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4415.390616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.024899] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4425.654677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4426.775845] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4433.889596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4433.893104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.909936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4435.940072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4435.942845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4435.944395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4435.977787] Lustre: Skipped 1 previous similar message [ 4441.060753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4446.175120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4446.180146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4446.205623] Lustre: Skipped 2 previous similar messages [ 4446.231134] LustreError: 165668:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4446.331956] Lustre: server umount lustre-OST0000 complete [ 4451.297798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.310688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4451.320614] Lustre: Skipped 1 previous similar message [ 4461.543186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4461.547275] Lustre: Skipped 3 previous similar messages [ 4464.095145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4464.174878] LustreError: 165873:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4464.185319] LustreError: 165873:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4464.261894] Lustre: server umount lustre-MDT0000 complete [ 4466.660197] LustreError: 164135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.669246] LustreError: 164135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4467.824163] LustreError: 163404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338030 with bad export cookie 6091093872488026153 [ 4467.832591] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4467.846070] LustreError: 163404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4468.012927] LustreError: 166076:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4468.017296] LustreError: 166076:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4468.238322] Lustre: server umount lustre-MDT0001 complete [ 4479.138403] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 4481.540432] Key type lgssc unregistered [ 4481.829675] LNet: 166609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4481.836870] LNetError: 166609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4481.865763] LNet: Removed LNI 192.168.202.143@tcp [ 4482.633341] Key type .llcrypt unregistered [ 4482.635023] Key type ._llcrypt unregistered [ 4503.597703] Key type ._llcrypt registered [ 4503.600529] Key type .llcrypt registered [ 4503.731454] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 4504.476851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4504.630137] alg: No test for adler32 (adler32-zlib) [ 4505.663305] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 4505.985752] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 4507.631185] Key type lgssc registered [ 4508.701155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4518.146169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4518.167835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4519.478124] Lustre: *** cfs_fail_loc=135, val=0*** [ 4519.537882] LustreError: 168113:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4519.549607] LustreError: 168113:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.143@tcp: cfg command failed: rc = -2 [ 4519.561206] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4519.561206] [ 4519.569404] LustreError: MGC192.168.202.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4519.581705] LustreError: 168073:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4519.595966] LustreError: 168073:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4519.608338] LustreError: 168073:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 4519.611031] LustreError: 168073:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGC192.168.202.143@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000014e6d39c) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4519.635196] LustreError: 168073:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4519.718607] Lustre: server umount lustre-MDT0000 complete [ 4519.720535] LustreError: 168073:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4525.317054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4525.698860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4528.541192] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4535.265527] LustreError: 168485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4535.274546] LustreError: 168485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4536.069901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4536.570794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4540.216572] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4545.881382] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4550.106063] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4555.243910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4555.258427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4555.260380] Lustre: Skipped 1 previous similar message [ 4555.273193] Lustre: Skipped 1 previous similar message [ 4558.923664] LustreError: 169829:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4558.927541] LustreError: 169829:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4559.055883] Lustre: server umount lustre-MDT0000 complete [ 4560.355349] LustreError: 168486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4560.384682] LustreError: 168486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4561.382681] LustreError: 168486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4561.403746] LustreError: 168486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4562.412661] LustreError: 168466:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338125 with bad export cookie 17532855147294344915 [ 4562.436852] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4562.574393] LustreError: 170030:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4562.578294] LustreError: 170030:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4562.759620] Lustre: server umount lustre-MDT0001 complete [ 4572.111169] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 4574.197169] Key type lgssc unregistered [ 4574.398619] LNet: 170560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4574.408122] LNetError: 170560:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4574.424588] LNet: Removed LNI 192.168.202.143@tcp [ 4575.009420] Key type .llcrypt unregistered [ 4575.011115] Key type ._llcrypt unregistered [ 4592.155404] Key type ._llcrypt registered [ 4592.157239] Key type .llcrypt registered [ 4592.243303] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 4592.478066] Key type .llcrypt unregistered [ 4592.480284] Key type ._llcrypt unregistered [ 4607.161294] Key type ._llcrypt registered [ 4607.162989] Key type .llcrypt registered [ 4607.267819] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 4608.048735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4608.073121] alg: No test for adler32 (adler32-zlib) [ 4609.058527] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 4609.284417] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 4610.935222] Key type lgssc registered [ 4611.739569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4618.675876] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4618.692525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4634.207935] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4639.327198] LustreError: 172643:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.143@tcp: failed processing log, type 4: rc = -110 [ 4665.951554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.353496] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4675.003418] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4680.392940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4680.832524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4680.883490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4684.629027] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4686.310063] LustreError: 172670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4686.340547] LustreError: 172670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4691.296556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4691.425889] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4691.584722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4694.692629] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4698.805455] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4701.309861] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4703.418861] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4716.513104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4716.516577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4716.543794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4717.026657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4717.046387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.612981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4720.615739] Lustre: Skipped 1 previous similar message [ 4725.734481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4725.740917] Lustre: Skipped 1 previous similar message [ 4730.335142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4730.392149] LustreError: 175415:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4730.492571] Lustre: server umount lustre-OST0000 complete [ 4735.972208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4735.979442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4735.983475] Lustre: Skipped 1 previous similar message [ 4739.589836] LustreError: 175618:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4739.608032] LustreError: 175618:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4739.710992] Lustre: server umount lustre-MDT0000 complete [ 4741.092796] LustreError: 173574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.108569] LustreError: 173574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4742.800448] LustreError: 172651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338305 with bad export cookie 11659427226534379332 [ 4742.809090] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4742.815051] LustreError: 172651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4742.908392] LustreError: 175820:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4742.910528] LustreError: 175820:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4743.117387] Lustre: server umount lustre-MDT0001 complete [ 4752.238212] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 4754.594477] Key type lgssc unregistered [ 4754.899963] LNet: 176352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4754.903677] LNetError: 176352:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4754.921090] LNet: Removed LNI 192.168.202.143@tcp [ 4755.854949] Key type .llcrypt unregistered [ 4755.857373] Key type ._llcrypt unregistered [ 4775.100977] Key type ._llcrypt registered [ 4775.106873] Key type .llcrypt registered [ 4775.195822] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 4775.824261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4775.857093] alg: No test for adler32 (adler32-zlib) [ 4776.910786] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 4777.096841] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 4778.752541] Key type lgssc registered [ 4779.474780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4785.628966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4785.646545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.068523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4790.660164] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4797.408773] LustreError: 177723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.424372] LustreError: 177723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4798.052428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4798.362366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4801.680264] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4806.275270] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4809.712230] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4815.701737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4816.022115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4820.763320] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4826.227210] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4832.445237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4839.963671] Lustre: Failing over lustre-MDT0000 [ 4840.054279] LustreError: 180044:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4840.175716] Lustre: server umount lustre-MDT0000 complete [ 4842.410618] LustreError: 177718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4842.977468] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4842.996276] Lustre: Skipped 2 previous similar messages [ 4847.511240] LustreError: 177720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.526390] LustreError: 177720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4852.634455] LustreError: 177718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4852.645802] LustreError: 177718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4856.765645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4856.879983] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4857.050129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4857.102057] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4857.758882] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4859.597668] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4862.442563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4862.462416] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4862.524778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4862.614596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4865.954161] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4872.682341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4872.688558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4876.037181] LustreError: 181229:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4876.044203] LustreError: 181229:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4876.130515] Lustre: server umount lustre-OST0000 complete [ 4882.915322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4882.918918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4882.923984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4882.924283] Lustre: Skipped 2 previous similar messages [ 4882.928827] Lustre: Skipped 1 previous similar message [ 4885.201615] LustreError: 181430:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4885.204664] LustreError: 181430:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4885.272914] Lustre: server umount lustre-MDT0000 complete [ 4888.032266] LustreError: 177719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4888.057538] LustreError: 177719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4888.074541] LustreError: 178442:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338451 with bad export cookie 13306744174735763129 [ 4888.079980] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4888.089216] LustreError: 178442:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4888.206602] LustreError: 181631:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4888.208710] LustreError: 181631:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4888.369511] Lustre: server umount lustre-MDT0001 complete [ 4897.744731] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 4899.956841] Key type lgssc unregistered [ 4900.144866] LNet: 182163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4900.158974] LNetError: 182163:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4900.169875] LNet: Removed LNI 192.168.202.143@tcp [ 4900.939675] Key type .llcrypt unregistered [ 4900.942067] Key type ._llcrypt unregistered [ 4921.493559] Key type ._llcrypt registered [ 4921.495289] Key type .llcrypt registered [ 4921.579984] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 4922.664064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4922.747804] alg: No test for adler32 (adler32-zlib) [ 4923.853445] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 4924.094067] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 4925.807225] Key type lgssc registered [ 4927.198646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4936.402453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4936.440389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4938.295102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4942.791882] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4948.458730] LustreError: 183556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4948.479707] LustreError: 183556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4951.230738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4951.584805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4955.519554] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4961.855939] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4964.894424] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4971.840259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4972.185926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4977.175567] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4982.204609] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4983.487630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4988.828161] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4991.809104] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5000.033855] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5014.496038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5014.501119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5014.511661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5014.535706] Lustre: Skipped 1 previous similar message [ 5014.545317] Lustre: Skipped 1 previous similar message [ 5018.946209] LustreError: 185994:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5019.095261] Lustre: server umount lustre-OST0000 complete [ 5023.202359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5023.206092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5023.216037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5024.736590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5024.743328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.483564] LustreError: 186196:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5028.487798] LustreError: 186196:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5028.565085] Lustre: server umount lustre-MDT0000 complete [ 5029.857535] LustreError: 183551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5029.879881] LustreError: 183551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5031.683371] LustreError: 183536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338594 with bad export cookie 17616566508061918588 [ 5031.688796] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5031.703580] LustreError: 183536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5031.831593] LustreError: 186398:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5031.836436] LustreError: 186398:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5031.969610] Lustre: server umount lustre-MDT0001 complete [ 5040.607335] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 5042.941657] Key type lgssc unregistered [ 5043.212785] LNet: 186930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5043.216961] LNetError: 186930:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5043.229921] LNet: Removed LNI 192.168.202.143@tcp [ 5044.051615] Key type .llcrypt unregistered [ 5044.053528] Key type ._llcrypt unregistered [ 5066.644252] Key type ._llcrypt registered [ 5066.645604] Key type .llcrypt registered [ 5066.787526] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 5068.036804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5068.073678] alg: No test for adler32 (adler32-zlib) [ 5069.121492] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 5069.363374] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 5071.023193] Key type lgssc registered [ 5072.166835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5079.456472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5079.496648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5081.042342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5084.508542] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5091.300281] LustreError: 188323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5091.319516] LustreError: 188323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5092.131426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5092.579298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5095.459709] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5099.924243] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5103.300056] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5110.191642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5110.666177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5115.940933] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5120.889707] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5126.031302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5131.313609] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5144.565833] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5155.868561] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5162.602740] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5169.120079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5169.130444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5169.155791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5172.196420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5172.197621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5172.212409] Lustre: Skipped 1 previous similar message [ 5174.273708] LustreError: 192061:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5174.504553] Lustre: server umount lustre-OST0000 complete [ 5179.875757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5179.881587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5179.909452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5182.433744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5182.443565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.388591] LustreError: 192263:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5184.391012] LustreError: 192263:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5184.643786] Lustre: server umount lustre-MDT0000 complete [ 5187.566673] LustreError: 188322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5187.608111] LustreError: 188322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5188.866811] LustreError: 191862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338752 with bad export cookie 689958583240167569 [ 5188.872423] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5188.891706] LustreError: 191862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5189.089816] LustreError: 192464:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5189.111839] LustreError: 192464:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5189.289643] Lustre: server umount lustre-MDT0001 complete [ 5197.230614] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 5199.248334] Key type lgssc unregistered [ 5199.463713] LNet: 192997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5199.472278] LNetError: 192997:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5199.486150] LNet: Removed LNI 192.168.202.143@tcp [ 5200.264780] Key type .llcrypt unregistered [ 5200.266291] Key type ._llcrypt unregistered [ 5217.883895] Key type ._llcrypt registered [ 5217.885096] Key type .llcrypt registered [ 5217.936537] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 5218.397933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5218.424116] alg: No test for adler32 (adler32-zlib) [ 5219.341263] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 5219.475663] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 5221.103184] Key type lgssc registered [ 5221.793748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5228.182790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5228.208684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.628377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5232.874553] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5239.779020] LustreError: 194371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5239.813898] LustreError: 194371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5240.306245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5240.656412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5243.612378] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5247.239956] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5250.267580] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5255.733281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5256.122951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.834773] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5265.238666] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5274.557319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5289.957798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5289.961127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5289.972839] Lustre: Skipped 1 previous similar message [ 5289.975170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5294.278118] LustreError: 196792:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5294.390216] Lustre: server umount lustre-OST0000 complete [ 5297.121972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5297.125632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.142528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5297.146187] Lustre: Skipped 1 previous similar message [ 5300.192936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5300.192982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.242234] LustreError: 196993:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5303.244652] LustreError: 196993:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5303.325484] Lustre: server umount lustre-MDT0000 complete [ 5305.312155] LustreError: 194370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5305.322196] LustreError: 194370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5306.381758] LustreError: 194350:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338869 with bad export cookie 9733011246170974781 [ 5306.388266] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5306.396988] LustreError: 194350:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5306.501691] LustreError: 197195:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5306.504390] LustreError: 197195:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5306.608051] Lustre: server umount lustre-MDT0001 complete [ 5315.779395] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 5318.115186] Key type lgssc unregistered [ 5318.358861] LNet: 197726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5318.364354] LNetError: 197726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5318.378522] LNet: Removed LNI 192.168.202.143@tcp [ 5319.010468] Key type .llcrypt unregistered [ 5319.012224] Key type ._llcrypt unregistered [ 5336.190657] Key type ._llcrypt registered [ 5336.192594] Key type .llcrypt registered [ 5336.287731] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 5337.457817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5337.479986] alg: No test for adler32 (adler32-zlib) [ 5338.454864] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 5338.728336] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 5340.431307] Key type lgssc registered [ 5341.567662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5348.295788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5348.308716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5349.779415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5353.324985] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5359.075983] LustreError: 198972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5359.099887] LustreError: 198972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5360.588549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5361.021277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5364.294462] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5369.276284] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5372.414164] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5378.785556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5379.241942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5384.683225] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5390.116980] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5394.675056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5407.200049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5407.209929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5407.222570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5410.274955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5410.281145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5410.303742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5410.328659] Lustre: Skipped 1 previous similar message [ 5412.949167] LustreError: 201370:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5413.214522] Lustre: server umount lustre-OST0000 complete [ 5417.448670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5417.456966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5417.478454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5420.513692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5420.522058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5423.412693] LustreError: 201571:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5423.424986] LustreError: 201571:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5423.550703] Lustre: server umount lustre-MDT0000 complete [ 5425.643689] LustreError: 198972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5425.684203] LustreError: 198972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5427.597549] LustreError: 198952:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338990 with bad export cookie 4556637043833909130 [ 5427.609085] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5427.619620] LustreError: 198952:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5427.788989] LustreError: 201773:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5427.792114] LustreError: 201773:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5428.067887] Lustre: server umount lustre-MDT0001 complete [ 5438.856702] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 5442.172381] Key type lgssc unregistered [ 5442.504909] LNet: 202304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5442.512965] LNetError: 202304:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5442.522765] LNet: Removed LNI 192.168.202.143@tcp [ 5443.317791] Key type .llcrypt unregistered [ 5443.321342] Key type ._llcrypt unregistered [ 5462.782033] Key type ._llcrypt registered [ 5462.784456] Key type .llcrypt registered [ 5462.894257] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 5463.720281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5463.811930] alg: No test for adler32 (adler32-zlib) [ 5464.948212] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 5465.218693] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 5466.855168] Key type lgssc registered [ 5467.972555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5474.194567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5474.221939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5475.603228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5478.704919] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5485.026989] LustreError: 203551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5485.038993] LustreError: 203551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5485.751079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5486.069887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5489.506133] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5494.880129] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5498.563195] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5504.476825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5504.781883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5510.641628] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5515.635098] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5520.182184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5528.251323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5528.294207] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5528.457433] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5533.552336] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5538.777260] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5540.300394] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5551.888118] Lustre: Permanently deactivating lustre-OST0001 [ 5551.900841] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5561.372606] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5568.758775] Lustre: Permanently reactivating lustre-OST0001 [ 5568.777917] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5568.785260] Lustre: Skipped 2 previous similar messages [ 5569.743908] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.773066] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5569.786958] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5569.796171] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5569.822982] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5578.723466] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.726885] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5578.738950] Lustre: Skipped 2 previous similar messages [ 5583.842757] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5583.850870] Lustre: Skipped 2 previous similar messages [ 5584.315153] LustreError: 207050:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5584.364921] Lustre: server umount lustre-OST0001 complete [ 5587.939469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5587.942705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.965806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5588.964950] LustreError: 205147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5588.974000] LustreError: 205147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5593.291190] LustreError: 207251:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5593.296188] LustreError: 207251:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5593.476984] Lustre: server umount lustre-OST0000 complete [ 5598.691560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5598.703992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5598.730335] Lustre: Skipped 1 previous similar message [ 5598.733608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5598.735776] Lustre: Skipped 2 previous similar messages [ 5603.361790] LustreError: 207452:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5603.364859] LustreError: 207452:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5603.472247] Lustre: server umount lustre-MDT0000 complete [ 5604.334140] LustreError: 203550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.708516] LustreError: 203532:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339170 with bad export cookie 9722803896851575758 [ 5607.715246] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5607.923201] LustreError: 207655:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5607.925276] LustreError: 207655:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5608.209356] Lustre: server umount lustre-MDT0001 complete [ 5620.730820] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 5623.086942] Key type lgssc unregistered [ 5623.416216] LNet: 208187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5623.432576] LNetError: 208187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5623.443685] LNet: Removed LNI 192.168.202.143@tcp [ 5624.111739] Key type .llcrypt unregistered [ 5624.114138] Key type ._llcrypt unregistered [ 5654.365899] Key type ._llcrypt registered [ 5654.367782] Key type .llcrypt registered [ 5654.481310] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 5667.865587] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 5669.091641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5669.158055] alg: No test for adler32 (adler32-zlib) [ 5670.262167] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 5670.524427] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 5672.207167] Key type lgssc registered [ 5673.317590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5681.827813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5688.368248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5693.990830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5700.099563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5706.628546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5706.676552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5706.687252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.000831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5708.042385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5708.122193] Lustre: lustre-MDT0000: new disk, initializing [ 5708.277949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5708.298138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5711.647116] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5720.872970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5720.915629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5720.983576] Lustre: 212134:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5721.059808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5721.067584] Lustre: Skipped 1 previous similar message [ 5721.136828] Lustre: lustre-MDT0001: new disk, initializing [ 5721.218458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5721.250574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5721.255414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5724.744229] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5732.558923] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5736.256365] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5743.478388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5743.552339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5743.746947] Lustre: lustre-OST0000: new disk, initializing [ 5743.749683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5743.829648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5749.352777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5749.379732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5749.410964] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5749.495589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5756.785096] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5761.388419] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5761.641773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5765.821519] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5766.062885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5770.211647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5770.214139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.222052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5770.284674] Lustre: Skipped 1 previous similar message [ 5773.971158] LustreError: 215073:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5774.145934] Lustre: server umount lustre-OST0000 complete [ 5780.448896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.454671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5780.469992] Lustre: Skipped 1 previous similar message [ 5780.492671] Lustre: Skipped 2 previous similar messages [ 5784.844219] LustreError: 215275:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5784.847267] LustreError: 215275:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5784.975068] Lustre: server umount lustre-MDT0000 complete [ 5788.739477] LustreError: 212127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339351 with bad export cookie 5765861209643398456 [ 5788.748392] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.757091] LustreError: 212127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5788.902259] LustreError: 215476:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5788.912753] LustreError: 215476:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5789.172843] Lustre: server umount lustre-MDT0001 complete [ 5797.539557] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 19:29:20 (1763339360) [ 5804.072579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5804.582058] LustreError: 216148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5804.606510] LustreError: 216148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5804.673575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5808.742994] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5810.147992] LustreError: 216148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5814.242451] LustreError: 216148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5816.738903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5817.148788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5820.335475] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5826.378302] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5830.417158] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5837.013057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5837.414406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.336467] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5848.944903] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5855.850320] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5858.263355] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5866.166825] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5876.138455] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5884.784324] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5905.731367] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5905.741073] Lustre: Skipped 2 previous similar messages [ 5944.390502] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5944.395357] Lustre: Skipped 4 previous similar messages [ 6017.675530] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6017.678778] Lustre: Skipped 8 previous similar messages [ 6024.164639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6024.170288] Lustre: Skipped 1 previous similar message [ 6024.172602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6024.179989] LustreError: Skipped 1 previous similar message [ 6024.186974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6024.194407] Lustre: Skipped 1 previous similar message [ 6029.111680] LustreError: 219639:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6029.120330] LustreError: 219639:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6029.238586] Lustre: server umount lustre-OST0000 complete [ 6034.405818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6034.412411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6034.422017] Lustre: Skipped 1 previous similar message [ 6034.436322] Lustre: Skipped 1 previous similar message [ 6039.320608] LustreError: 219845:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6039.325570] LustreError: 219845:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6039.415297] Lustre: server umount lustre-MDT0000 complete [ 6039.525595] LustreError: 216143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6039.537250] LustreError: 216143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6042.786392] LustreError: 216130:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339605 with bad export cookie 5765861209643399338 [ 6042.789193] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6042.801118] LustreError: 216130:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6043.146765] Lustre: server umount lustre-MDT0001 complete [ 6052.566582] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 6055.457359] Key type lgssc unregistered [ 6055.776403] LNet: 220578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6055.780740] LNetError: 220578:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6056.804721] LNet: Removed LNI 192.168.202.143@tcp [ 6057.777899] Key type .llcrypt unregistered [ 6057.785784] Key type ._llcrypt unregistered [ 6077.928770] Key type ._llcrypt registered [ 6077.934508] Key type .llcrypt registered [ 6078.025098] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 6078.929444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6078.949938] alg: No test for adler32 (adler32-zlib) [ 6079.987587] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 6080.176402] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 6081.815468] Key type lgssc registered [ 6083.070653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6090.736217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6090.749443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.271923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6096.270095] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6102.498397] LustreError: 221951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6102.517623] LustreError: 221951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6104.650865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6104.991601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6109.117895] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6114.956387] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6119.155960] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6126.305203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6126.761857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6132.984645] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6138.149692] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6156.769115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6156.772416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.782184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6157.796576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6157.800387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.806270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6157.823636] Lustre: Skipped 1 previous similar message [ 6158.423144] LustreError: 224354:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6158.616358] Lustre: server umount lustre-OST0000 complete [ 6162.912735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.915608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6162.939163] Lustre: Skipped 1 previous similar message [ 6162.956362] Lustre: Skipped 1 previous similar message [ 6168.036342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6168.042705] Lustre: Skipped 1 previous similar message [ 6168.233917] LustreError: 224556:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6168.247730] LustreError: 224556:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6168.375238] Lustre: server umount lustre-MDT0000 complete [ 6171.597170] LustreError: 221931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339734 with bad export cookie 737869126315808146 [ 6171.600986] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6171.612958] LustreError: 221931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6171.720505] LustreError: 224759:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6171.734618] LustreError: 224759:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6171.838832] Lustre: server umount lustre-MDT0001 complete [ 6181.963988] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 6185.818282] Key type lgssc unregistered [ 6186.081725] LNet: 225291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6186.094124] LNetError: 225291:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6186.117344] LNet: Removed LNI 192.168.202.143@tcp [ 6187.169402] Key type .llcrypt unregistered [ 6187.173776] Key type ._llcrypt unregistered [ 6211.482084] Key type ._llcrypt registered [ 6211.483743] Key type .llcrypt registered [ 6211.599764] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 6211.991418] Key type .llcrypt unregistered [ 6211.995597] Key type ._llcrypt unregistered [ 6238.000499] Key type ._llcrypt registered [ 6238.002195] Key type .llcrypt registered [ 6238.086756] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 6255.794758] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 6257.270261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6257.307421] alg: No test for adler32 (adler32-zlib) [ 6258.552935] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 6258.873419] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 6260.631180] Key type lgssc registered [ 6262.194389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6271.837365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6279.348325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6286.319941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6292.794919] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6294.596680] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6053 sec ========== 19:37:37 (1763339857) [ 6296.294652] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:37:39 (1763339859) === [ 6299.615942] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:37:42 (1763339862) === [ 6339.745821] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 6342.000576] Key type lgssc unregistered [ 6342.336785] LNet: 231406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6342.356409] LNetError: 231406:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6342.381561] LNet: Removed LNI 192.168.202.143@tcp [ 6343.375296] Key type .llcrypt unregistered [ 6343.376918] Key type ._llcrypt unregistered