[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498716914 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003178] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005008] kvm-guest: setup PV IPIs [ 0.007421] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008016] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.012072] Yama: becoming mindful. [ 0.013028] SELinux: Initializing. [ 0.014084] *** VALIDATE selinux *** [ 0.022119] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027357] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029095] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030205] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031097] *** VALIDATE tmpfs *** [ 0.033087] *** VALIDATE proc *** [ 0.034196] *** VALIDATE cgroup *** [ 0.035006] *** VALIDATE cgroup2 *** [ 0.036272] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037132] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039022] Spectre V2 : User space: Vulnerable [ 0.040005] Speculative Store Bypass: Vulnerable [ 0.043412] debug: unmapping init [mem 0xffffffff8fe59000-0xffffffff8fe60fff] [ 0.045909] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046575] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047015] ... version: 2 [ 0.048006] ... bit width: 48 [ 0.049007] ... generic registers: 4 [ 0.050008] ... value mask: 0000ffffffffffff [ 0.051008] ... max period: 00007fffffffffff [ 0.052007] ... fixed-purpose events: 3 [ 0.053007] ... event mask: 000000070000000f [ 0.054236] rcu: Hierarchical SRCU implementation. [ 0.056340] smp: Bringing up secondary CPUs ... [ 0.057500] x86: Booting SMP configuration: [ 0.058015] .... node #0, CPUs: #1 #2 #3 [ 0.060682] smp: Brought up 1 node, 4 CPUs [ 0.062008] smpboot: Max logical packages: 1 [ 0.063008] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.230057] node 0 deferred pages initialised in 166ms [ 0.233110] devtmpfs: initialized [ 0.234183] x86/mm: Memory block size: 128MB [ 0.236427] gcov: version magic: 0x41383552 [ 0.238197] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.241070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243277] pinctrl core: initialized pinctrl subsystem [ 0.245126] [ 0.245579] ************************************************************* [ 0.247012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249017] ** ** [ 0.251010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.252008] ** ** [ 0.253012] ** This means that this kernel is built to expose internal ** [ 0.255010] ** IOMMU data structures, which may compromise security on ** [ 0.257011] ** your system. ** [ 0.258008] ** ** [ 0.260010] ** If you see this message and you are not debugging the ** [ 0.262011] ** kernel, report this immediately to your vendor! ** [ 0.263010] ** ** [ 0.265026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267019] ************************************************************* [ 0.269140] NET: Registered protocol family 16 [ 0.270439] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.271064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.273045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.276119] cpuidle: using governor menu [ 0.278000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.280465] PCI: Using configuration type 1 for base access [ 0.282130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.292034] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.293016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.295051] cryptd: max_cpu_qlen set to 1000 [ 0.298253] ACPI: Added _OSI(Module Device) [ 0.299156] ACPI: Added _OSI(Processor Device) [ 0.300008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.301009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.304576] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.309582] ACPI: Interpreter enabled [ 0.310050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.311013] ACPI: Using IOAPIC for interrupt routing [ 0.312133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.314355] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.323044] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.324025] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.326012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.327078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.331201] acpiphp: Slot [2] registered [ 0.332075] acpiphp: Slot [3] registered [ 0.333072] acpiphp: Slot [4] registered [ 0.334064] acpiphp: Slot [5] registered [ 0.335091] acpiphp: Slot [6] registered [ 0.336066] acpiphp: Slot [7] registered [ 0.337020] acpiphp: Slot [8] registered [ 0.338076] acpiphp: Slot [9] registered [ 0.339012] acpiphp: Slot [10] registered [ 0.339977] acpiphp: Slot [11] registered [ 0.341052] acpiphp: Slot [12] registered [ 0.342083] acpiphp: Slot [13] registered [ 0.343000] acpiphp: Slot [14] registered [ 0.343000] acpiphp: Slot [15] registered [ 0.343000] acpiphp: Slot [16] registered [ 0.345054] acpiphp: Slot [17] registered [ 0.346055] acpiphp: Slot [18] registered [ 0.347061] acpiphp: Slot [19] registered [ 0.348043] acpiphp: Slot [20] registered [ 0.349017] acpiphp: Slot [21] registered [ 0.349992] acpiphp: Slot [22] registered [ 0.350055] acpiphp: Slot [23] registered [ 0.351039] acpiphp: Slot [24] registered [ 0.352021] acpiphp: Slot [25] registered [ 0.353023] acpiphp: Slot [26] registered [ 0.353981] acpiphp: Slot [27] registered [ 0.355097] acpiphp: Slot [28] registered [ 0.356056] acpiphp: Slot [29] registered [ 0.357057] acpiphp: Slot [30] registered [ 0.358093] acpiphp: Slot [31] registered [ 0.359045] PCI host bridge to bus 0000:00 [ 0.360011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.361017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.363016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.365014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.366013] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.368022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.369163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.373209] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.376268] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.384016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.387933] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.390023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.392011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.394022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.396550] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.399627] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.401019] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.402727] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.407015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.418890] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.424021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.429415] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.444026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.452020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.473151] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.489438] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.502021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.509022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.532022] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.549447] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.559016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.570045] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.598023] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.615820] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.621015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.629016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.646020] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.656660] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.669029] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.678029] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.704016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.713000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.721021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.726019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.744031] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.754211] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.757419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.759432] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.761407] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.764263] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.769285] iommu: Default domain type: Passthrough [ 0.771461] SCSI subsystem initialized [ 0.773124] ACPI: bus type USB registered [ 0.774077] usbcore: registered new interface driver usbfs [ 0.776079] usbcore: registered new interface driver hub [ 0.778101] usbcore: registered new device driver usb [ 0.779111] pps_core: LinuxPPS API ver. 1 registered [ 0.781012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.783079] PTP clock support registered [ 0.785077] EDAC MC: Ver: 3.0.0 [ 0.787338] PCI: Using ACPI for IRQ routing [ 0.789000] NetLabel: Initializing [ 0.791016] NetLabel: domain hash size = 128 [ 0.792005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.794101] NetLabel: unlabeled traffic allowed by default [ 0.797270] vgaarb: loaded [ 0.798433] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.800019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.805000] clocksource: Switched to clocksource kvm-clock [ 0.916603] VFS: Disk quotas dquot_6.6.0 [ 0.918362] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.920246] *** VALIDATE ramfs *** [ 0.921239] *** VALIDATE hugetlbfs *** [ 0.922518] pnp: PnP ACPI init [ 0.924660] pnp: PnP ACPI: found 6 devices [ 0.941528] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.943946] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.945461] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.947152] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.949019] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.950883] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.953018] NET: Registered protocol family 2 [ 0.954902] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.958977] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.962069] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.966304] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.969220] TCP: Hash tables configured (established 65536 bind 65536) [ 0.971471] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.973914] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.976752] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.979217] NET: Registered protocol family 1 [ 0.981117] RPC: Registered named UNIX socket transport module. [ 0.982685] RPC: Registered udp transport module. [ 0.983904] RPC: Registered tcp transport module. [ 0.985150] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.986807] NET: Registered protocol family 44 [ 0.987916] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.989526] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.991086] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.992870] PCI: CLS 0 bytes, default 64 [ 0.994629] Unpacking initramfs... [ 2.495716] debug: unmapping init [mem 0xffff99d8fcc54000-0xffff99d8fffbffff] [ 2.499319] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.501675] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.504797] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.024334] Initialise system trusted keyrings [ 3.025581] Key type blacklist registered [ 3.027627] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.035549] zbud: loaded [ 3.037937] *** VALIDATE nfs *** [ 3.038889] *** VALIDATE nfs4 *** [ 3.041197] pstore: using deflate compression [ 3.046886] Platform Keyring initialized [ 3.213157] NET: Registered protocol family 38 [ 3.214535] Key type asymmetric registered [ 3.215751] Asymmetric key parser 'x509' registered [ 3.218136] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.221467] io scheduler mq-deadline registered [ 3.222965] io scheduler kyber registered [ 3.224427] io scheduler bfq registered [ 3.226161] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.228871] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.231482] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.233967] ACPI: Power Button [PWRF] [ 3.341620] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.457455] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.686272] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.809148] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.046470] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.074550] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.107364] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.111782] Non-volatile memory driver v1.3 [ 4.113107] Linux agpgart interface v0.103 [ 4.145328] virtio_blk virtio1: [vda] 67960 512-byte logical blocks (34.8 MB/33.2 MiB) [ 4.148973] vda: detected capacity change from 0 to 34795520 [ 4.179623] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.187393] vdb: detected capacity change from 0 to 1073741824 [ 4.206731] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.209870] vdc: detected capacity change from 0 to 2621440000 [ 4.227713] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.230239] vdd: detected capacity change from 0 to 2621440000 [ 4.247472] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.251437] vde: detected capacity change from 0 to 4294967296 [ 4.271834] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.274503] vdf: detected capacity change from 0 to 4294967296 [ 4.283921] libphy: Fixed MDIO Bus: probed [ 4.293321] usbcore: registered new interface driver usbserial_generic [ 4.297921] usbserial: USB Serial support registered for generic [ 4.300260] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.303755] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.305770] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.308627] mousedev: PS/2 mouse device common for all mice [ 4.312985] rtc_cmos 00:05: RTC can wake from S4 [ 4.316445] rtc_cmos 00:05: registered as rtc0 [ 4.318191] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.320925] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.321560] intel_pstate: CPU model not supported [ 4.330145] hid: raw HID events driver (C) Jiri Kosina [ 4.330201] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.332978] usbcore: registered new interface driver usbhid [ 4.337771] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.338432] usbhid: USB HID core driver [ 4.338592] drop_monitor: Initializing network drop monitor service [ 4.338711] Initializing XFRM netlink socket [ 4.347548] NET: Registered protocol family 10 [ 4.350774] Segment Routing with IPv6 [ 4.352332] NET: Registered protocol family 17 [ 4.354119] mpls_gso: MPLS GSO support [ 4.363588] RAS: Correctable Errors collector initialized. [ 4.366079] AVX version of gcm_enc/dec engaged. [ 4.368178] AES CTR mode by8 optimization enabled [ 4.493300] sched_clock: Marking stable (4493190325, 0)->(5435121205, -941930880) [ 4.500454] registered taskstats version 1 [ 4.503582] Loading compiled-in X.509 certificates [ 4.509701] zswap: loaded using pool lzo/zbud [ 4.549713] Key type big_key registered [ 4.592603] Key type encrypted registered [ 4.594793] ima: No TPM chip found, activating TPM-bypass! [ 4.602487] ima: Allocated hash algorithm: sha1 [ 4.604033] ima: No architecture policies found [ 4.605937] evm: Initialising EVM extended attributes: [ 4.608394] evm: security.selinux [ 4.609513] evm: security.ima [ 4.610681] evm: security.capability [ 4.612109] evm: HMAC attrs: 0x1 [ 4.615651] rtc_cmos 00:05: setting system clock to 2025-10-07 13:10:23 UTC (1759842623) [ 4.625740] debug: unmapping init [mem 0xffffffff90e03000-0xffffffff90ffffff] [ 4.628909] debug: unmapping init [mem 0xffffffff8fb82000-0xffffffff8fe58fff] [ 4.637143] Write protecting the kernel read-only data: 28672k [ 4.640502] debug: unmapping init [mem 0xffffffff8e203000-0xffffffff8e3fffff] [ 4.643231] debug: unmapping init [mem 0xffffffff8eb14000-0xffffffff8ebfffff] [ 4.686650] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.694801] systemd[1]: Detected virtualization kvm. [ 4.696758] systemd[1]: Detected architecture x86-64. [ 4.698729] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.729367] systemd[1]: No hostname configured. [ 4.730894] systemd[1]: Set hostname to . [ 4.732643] random: systemd: uninitialized urandom read (16 bytes read) [ 4.734759] systemd[1]: Initializing machine ID from random generator. [ 4.815485] random: ln: uninitialized urandom read (6 bytes read) [ 4.941389] random: systemd: uninitialized urandom read (16 bytes read) [ 4.944074] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.954249] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.970297] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.161792] device-mapper: uevent: version 1.0.3 [ 6.165091] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.601945] virtio_net virtio0 ens2: renamed from eth0 [ 7.633702] random: fast init done [ 7.798880] scsi host0: ata_piix [ 7.811371] scsi host1: ata_piix [ 7.812954] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.823596] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.124787] random: crng init done [ 13.126288] random: 7 urandom warning(s) missed due to ratelimiting [ 14.079196] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.495589] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.070215] printk: systemd: 25 output lines suppressed due to ratelimiting [ 17.781546] SELinux: Disabled at runtime. [ 17.893756] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.902807] systemd[1]: Detected virtualization kvm. [ 17.904671] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.847434] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.850355] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.867094] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.873134] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.879786] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.892808] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.899958] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ 19.003464] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.519962] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.157229] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.242639] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.322494] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.439460] EDAC sbridge: Ver: 1.1.2 [ 23.036977] Key type dns_resolver registered [ 23.652214] NFS: Registering the id_resolver key type [ 23.653525] Key type id_resolver registered [ 23.656137] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg355-server login: [ 45.599941] libcfs: loading out-of-tree module taints kernel. [ 45.615021] alg: No test for adler32 (adler32-zlib) [ 46.366288] Key type ._llcrypt registered [ 46.367119] Key type .llcrypt registered [ 46.398388] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 52.763096] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 53.219383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.420742] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 53.635508] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 53.637822] LNet: Accept secure, port 988 [ 55.231071] Key type lgssc registered [ 55.642600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 61.221826] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 61.996681] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 65.055414] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 67.239556] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 69.573498] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 74.208080] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 78.438194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 78.466394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 79.550522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 79.561175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 79.595377] Lustre: lustre-MDT0000: new disk, initializing [ 79.620275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 79.626051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 80.948540] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 85.782312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 85.805372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.829078] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 85.836189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 85.837234] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 85.839164] Lustre: Skipped 1 previous similar message [ 85.875562] Lustre: lustre-MDT0001: new disk, initializing [ 85.893385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 85.900683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 85.903570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 87.107475] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 92.421391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.450448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 92.453287] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 92.529782] Lustre: lustre-OST0000: new disk, initializing [ 92.531436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 92.547132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.955655] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 94.058544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 94.061448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 99.048991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.076286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 99.079396] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 99.114075] Lustre: lustre-OST0001: new disk, initializing [ 99.115749] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 99.132149] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 100.448798] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 102.827040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 102.829962] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 106.035439] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 109.464815] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 115.667660] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing check_logdir /tmp/testlogs/ [ 117.024740] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing yml_node [ 118.380204] Lustre: DEBUG MARKER: Client: 2.15.7.3 [ 119.175612] Lustre: DEBUG MARKER: MDS: 2.15.7.3 [ 119.958317] Lustre: DEBUG MARKER: OSS: 2.15.7.3 [ 120.518717] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Oct 7 09:12:18 EDT 2025 [ 123.326285] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball 110 [ 123.869428] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 137.183468] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 137.186749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 137.191800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 138.719849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 138.719900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 138.721841] Lustre: Skipped 2 previous similar messages [ 138.728078] Lustre: Skipped 2 previous similar messages [ 141.629145] Lustre: server umount lustre-MDT0000 complete [ 142.862581] LustreError: 8221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759842761 with bad export cookie 16335894302811213800 [ 142.863688] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 142.867171] LustreError: 8221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 143.839546] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 143.839724] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 143.839949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 143.839954] Lustre: Skipped 1 previous similar message [ 143.843841] LustreError: Skipped 1 previous similar message [ 143.847491] Lustre: Skipped 1 previous similar message [ 147.871476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 147.871545] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 147.873015] Lustre: Skipped 2 previous similar messages [ 147.876182] LustreError: Skipped 1 previous similar message [ 148.895745] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 148.899953] LustreError: Skipped 1 previous similar message [ 154.079936] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 154.084102] LustreError: Skipped 1 previous similar message [ 157.151106] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 157.225424] Lustre: server umount lustre-MDT0001 complete [ 164.585207] Lustre: server umount lustre-OST0000 complete [ 180.191102] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 180.242795] Lustre: server umount lustre-OST0001 complete [ 182.335936] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 185.108254] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 188.451776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.717692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.757244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.935784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 197.386759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 197.415933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.505352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 197.517131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 197.554960] Lustre: lustre-MDT0000: new disk, initializing [ 197.583110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.591732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.085586] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 203.306204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 203.331349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.368827] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 203.470934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 204.935218] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 208.855610] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 210.857357] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 213.185476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 213.213625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.217045] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 213.310247] Lustre: lustre-OST0000: new disk, initializing [ 213.312479] Lustre: Skipped 1 previous similar message [ 213.314852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 213.318199] Lustre: Skipped 2 previous similar messages [ 213.344985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.347862] Lustre: Skipped 1 previous similar message [ 214.750070] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 215.351098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 215.354186] Lustre: Skipped 1 previous similar message [ 215.356116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.600232] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 220.088896] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 220.163395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 221.598090] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 221.678934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 225.762690] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 225.763033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 225.765533] LustreError: Skipped 1 previous similar message [ 225.770874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 225.773621] Lustre: Skipped 5 previous similar messages [ 228.435541] Lustre: server umount lustre-OST0000 complete [ 230.880945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.884690] Lustre: Skipped 2 previous similar messages [ 244.191119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 244.226750] Lustre: server umount lustre-MDT0000 complete [ 245.471164] LustreError: 14299:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759842864 with bad export cookie 16335894302811215228 [ 245.473480] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 245.474938] LustreError: 14299:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 246.783961] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:14:24 (1759842864) [ 249.093094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 249.263800] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 249.288527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.781836] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 253.617573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 255.340714] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 257.910586] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 259.916329] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 262.210954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 263.776757] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 266.299051] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 273.888529] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 273.889939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.891625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 273.891632] Lustre: Skipped 6 previous similar messages [ 273.898958] Lustre: Skipped 1 previous similar message [ 278.994122] Lustre: server umount lustre-OST0000 complete [ 278.996060] Lustre: Skipped 1 previous similar message [ 287.613054] LustreError: 18691:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759842906 with bad export cookie 16335894302811216082 [ 287.615289] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 287.616523] LustreError: 18691:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 292.380691] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 293.386677] Key type lgssc unregistered [ 293.503387] LNet: 22361:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.506375] LNet: Removed LNI 192.168.203.155@tcp [ 293.816439] Key type .llcrypt unregistered [ 293.817747] Key type ._llcrypt unregistered [ 302.268157] alg: No test for adler32 (adler32-zlib) [ 303.019369] Key type ._llcrypt registered [ 303.020295] Key type .llcrypt registered [ 303.054844] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 303.355880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 303.583545] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 303.652214] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 303.653992] LNet: Accept secure, port 988 [ 305.231092] Key type lgssc registered [ 305.537924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 308.086853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 309.226230] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.240976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 310.583412] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 313.348149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 313.475276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 314.876591] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 317.243899] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 319.090337] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 321.273344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 321.378287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 322.687519] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 324.454666] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 325.016782] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 334.815971] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 334.819247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.824695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 338.912762] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 338.914801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 338.915281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.822237] Lustre: server umount lustre-OST0000 complete [ 344.032329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.033109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 344.037038] Lustre: Skipped 1 previous similar message [ 344.038925] Lustre: Skipped 1 previous similar message [ 349.151792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.153726] Lustre: Skipped 2 previous similar messages [ 354.272034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.273992] Lustre: Skipped 1 previous similar message [ 356.319082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 356.355412] Lustre: server umount lustre-MDT0000 complete [ 357.634915] LustreError: 27017:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759842976 with bad export cookie 3332368738996027325 [ 357.637711] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.638876] LustreError: 27017:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 357.751652] Lustre: server umount lustre-MDT0001 complete [ 362.399026] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 363.414594] Key type lgssc unregistered [ 363.528455] LNet: 27822:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.531673] LNet: Removed LNI 192.168.203.155@tcp [ 363.830099] Key type .llcrypt unregistered [ 363.831495] Key type ._llcrypt unregistered [ 372.505837] alg: No test for adler32 (adler32-zlib) [ 373.258357] Key type ._llcrypt registered [ 373.259870] Key type .llcrypt registered [ 373.293629] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 373.619037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 373.867272] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 373.944137] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 373.946460] LNet: Accept secure, port 988 [ 375.519121] Key type lgssc registered [ 375.880037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 378.566995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.715717] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.731635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 381.194103] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 384.192211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.318544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 385.803986] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 388.479956] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 390.463869] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 394.950295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 395.056311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 396.068909] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 396.393567] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 398.901130] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 401.375654] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 401.378117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.382720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 404.450534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 404.450860] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 404.456589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 407.256290] Lustre: server umount lustre-OST0000 complete [ 409.568059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 409.568719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 409.571897] Lustre: Skipped 1 previous similar message [ 409.575852] Lustre: Skipped 2 previous similar messages [ 414.688067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 414.689753] Lustre: Skipped 1 previous similar message [ 414.722020] Lustre: server umount lustre-MDT0000 complete [ 415.935824] LustreError: 29261:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843034 with bad export cookie 15873978538578315637 [ 415.938100] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 415.940389] LustreError: 29261:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 416.058330] Lustre: server umount lustre-MDT0001 complete [ 420.576474] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 421.573095] Key type lgssc unregistered [ 421.680240] LNet: 33274:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.683123] LNet: Removed LNI 192.168.203.155@tcp [ 421.976247] Key type .llcrypt unregistered [ 421.977490] Key type ._llcrypt unregistered [ 430.289504] alg: No test for adler32 (adler32-zlib) [ 431.040257] Key type ._llcrypt registered [ 431.041198] Key type .llcrypt registered [ 431.078922] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 431.504811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 431.697094] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 431.775200] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 431.777457] LNet: Accept secure, port 988 [ 433.351110] Key type lgssc registered [ 433.665555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 436.132147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.255038] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 437.269552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.610891] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 441.247374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.349379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 442.570179] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 444.850435] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 446.566376] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 448.632394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 448.730060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.076515] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 452.457488] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 452.835801] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 462.304512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.304688] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 462.309584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.313142] Lustre: Skipped 1 previous similar message [ 467.028770] Lustre: server umount lustre-OST0000 complete [ 472.031804] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 472.034104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.037323] Lustre: Skipped 1 previous similar message [ 472.039480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 474.308269] Lustre: server umount lustre-MDT0000 complete [ 475.506054] LustreError: 34712:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843094 with bad export cookie 17487595237505055050 [ 475.508304] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.511157] LustreError: 34712:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 475.624107] Lustre: server umount lustre-MDT0001 complete [ 480.094361] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 481.020563] Key type lgssc unregistered [ 481.130297] LNet: 38383:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.132609] LNet: Removed LNI 192.168.203.155@tcp [ 481.409975] Key type .llcrypt unregistered [ 481.411955] Key type ._llcrypt unregistered [ 489.783217] alg: No test for adler32 (adler32-zlib) [ 490.535531] Key type ._llcrypt registered [ 490.536595] Key type .llcrypt registered [ 490.571184] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 490.967923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 491.082667] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 491.152467] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 491.154228] LNet: Accept secure, port 988 [ 492.735130] Key type lgssc registered [ 493.062724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 495.562586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.691204] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.705933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.081446] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 500.804929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.921277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 502.251539] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 504.575986] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 506.371805] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 508.581401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 508.687842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.106643] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 511.784856] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 512.437780] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 521.185045] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 521.186734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.187833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 521.190265] Lustre: Skipped 1 previous similar message [ 521.194664] Lustre: Skipped 1 previous similar message [ 524.171767] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 525.802685] Lustre: server umount lustre-OST0000 complete [ 542.175772] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 542.178637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.184316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.027502] Lustre: server umount lustre-MDT0000 complete [ 546.326379] LustreError: 39823:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843165 with bad export cookie 4107221361455962371 [ 546.329723] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 546.330491] LustreError: 39823:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 552.508449] Lustre: server umount lustre-MDT0001 complete [ 557.350364] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 558.354798] Key type lgssc unregistered [ 558.474360] LNet: 43672:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.478338] LNet: Removed LNI 192.168.203.155@tcp [ 558.796596] Key type .llcrypt unregistered [ 558.797645] Key type ._llcrypt unregistered [ 567.407943] alg: No test for adler32 (adler32-zlib) [ 568.159405] Key type ._llcrypt registered [ 568.160500] Key type .llcrypt registered [ 568.200781] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 568.591069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 568.864399] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 568.952186] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 568.954494] LNet: Accept secure, port 988 [ 570.535115] Key type lgssc registered [ 570.906980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 573.629676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.786829] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.801843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.324086] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 579.179952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.295727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 580.722633] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 583.330214] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 585.097727] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 587.319965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 587.432701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 588.899931] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 590.501085] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 591.355218] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 600.033093] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.033507] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 600.034135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 600.037216] Lustre: Skipped 1 previous similar message [ 603.016783] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 603.019249] Lustre: Skipped 2 previous similar messages [ 604.805299] Lustre: server umount lustre-MDT0000 complete [ 605.152509] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.156482] LustreError: Skipped 2 previous similar messages [ 606.067902] LustreError: 47080:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843224 with bad export cookie 3684863012252490282 [ 606.070738] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 606.071723] LustreError: 47080:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 608.132096] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.141359] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 610.272128] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.272177] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.282022] Lustre: Skipped 1 previous similar message [ 610.283720] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 612.277769] Lustre: server umount lustre-MDT0001 complete [ 630.233784] Lustre: server umount lustre-OST0000 complete [ 636.986720] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 637.958395] Key type lgssc unregistered [ 638.060342] LNet: 49110:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.062966] LNet: Removed LNI 192.168.203.155@tcp [ 638.358686] Key type .llcrypt unregistered [ 638.360181] Key type ._llcrypt unregistered [ 647.554171] alg: No test for adler32 (adler32-zlib) [ 648.305435] Key type ._llcrypt registered [ 648.306573] Key type .llcrypt registered [ 648.338377] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 648.652901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 648.887694] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 648.968742] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 648.970742] LNet: Accept secure, port 988 [ 650.543090] Key type lgssc registered [ 650.861153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 653.342468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 670.815292] LustreError: 50544:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 1: rc = -5 [ 701.535171] LustreError: 50544:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 731.231189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 732.535317] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 734.870349] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 772.694317] Lustre: server umount lustre-OST0000 complete [ 778.735238] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 779.608685] Key type lgssc unregistered [ 779.700316] LNet: 52212:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.702751] LNet: Removed LNI 192.168.203.155@tcp [ 779.969154] Key type .llcrypt unregistered [ 779.970447] Key type ._llcrypt unregistered [ 788.045997] alg: No test for adler32 (adler32-zlib) [ 788.797297] Key type ._llcrypt registered [ 788.798447] Key type .llcrypt registered [ 788.828898] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 789.125910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.316553] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 789.383423] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 789.384846] LNet: Accept secure, port 988 [ 790.959096] Key type lgssc registered [ 791.245720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 793.641571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.761172] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 794.772869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 796.037737] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 798.614713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.711186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 799.962798] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 802.222586] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 803.938608] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 805.982647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 806.080951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.355692] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 809.193430] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 809.643586] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 813.281790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 813.283444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 816.485539] Lustre: server umount lustre-OST0000 complete [ 818.656804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 818.657881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 818.662885] Lustre: Skipped 1 previous similar message [ 818.666629] Lustre: Skipped 2 previous similar messages [ 823.775898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 823.778489] Lustre: Skipped 1 previous similar message [ 823.875078] Lustre: server umount lustre-MDT0000 complete [ 825.139031] LustreError: 53651:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843444 with bad export cookie 157004532725326871 [ 825.141683] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 825.142288] LustreError: 53651:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 825.254275] Lustre: server umount lustre-MDT0001 complete [ 829.830247] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 830.758353] Key type lgssc unregistered [ 830.866337] LNet: 57319:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.869179] LNet: Removed LNI 192.168.203.155@tcp [ 831.159488] Key type .llcrypt unregistered [ 831.160477] Key type ._llcrypt unregistered [ 839.741664] alg: No test for adler32 (adler32-zlib) [ 840.492436] Key type ._llcrypt registered [ 840.493642] Key type .llcrypt registered [ 840.538615] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 841.100955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 841.249896] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 841.335479] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 841.337218] LNet: Accept secure, port 988 [ 842.919111] Key type lgssc registered [ 843.284850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 846.033472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 863.519314] LustreError: 58754:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 1: rc = -5 [ 894.239278] LustreError: 58754:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 924.959204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 926.381362] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 928.896059] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 931.080400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.610761] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.638091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.660107] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 948.086080] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 951.008393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.144650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 952.621916] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 955.112728] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 956.955068] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 961.506539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 961.506799] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 961.507716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 961.512174] Lustre: Skipped 1 previous similar message [ 963.751717] Lustre: server umount lustre-OST0000 complete [ 977.888285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.889271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 977.891089] Lustre: Skipped 1 previous similar message [ 977.892484] Lustre: Skipped 1 previous similar message [ 983.007861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.009470] Lustre: Skipped 2 previous similar messages [ 988.127935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 988.129692] Lustre: Skipped 1 previous similar message [ 992.223113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 992.272929] Lustre: server umount lustre-MDT0000 complete [ 993.247833] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 993.253508] LustreError: Skipped 1 previous similar message [ 993.511872] LustreError: 58761:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843612 with bad export cookie 5291599112925625816 [ 993.514907] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 993.515112] LustreError: 58761:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 999.683804] Lustre: server umount lustre-MDT0001 complete [ 1004.549325] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1005.585966] Key type lgssc unregistered [ 1005.712410] LNet: 62625:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.714641] LNet: Removed LNI 192.168.203.155@tcp [ 1006.030560] Key type .llcrypt unregistered [ 1006.032338] Key type ._llcrypt unregistered [ 1014.635064] alg: No test for adler32 (adler32-zlib) [ 1015.386492] Key type ._llcrypt registered [ 1015.387829] Key type .llcrypt registered [ 1015.434482] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1015.820360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1016.026924] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1016.108813] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1016.111286] LNet: Accept secure, port 988 [ 1017.703413] Key type lgssc registered [ 1018.219155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1023.213367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.530969] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.570260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.955129] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1029.601206] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1032.232181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.435781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1034.936623] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1040.335820] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1044.468060] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1048.383061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1048.536629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.015214] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1052.295473] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1054.692395] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 1068.002234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.004030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1068.006832] Lustre: Skipped 1 previous similar message [ 1068.011676] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1071.393357] Lustre: server umount lustre-OST0000 complete [ 1073.120233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1073.120877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1073.124741] Lustre: Skipped 1 previous similar message [ 1073.127203] Lustre: Skipped 1 previous similar message [ 1078.240640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1078.243469] Lustre: Skipped 1 previous similar message [ 1078.925181] Lustre: server umount lustre-MDT0000 complete [ 1080.373268] LustreError: 64065:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843699 with bad export cookie 7725166244936810820 [ 1080.376733] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1080.378416] LustreError: 64065:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1080.516231] Lustre: server umount lustre-MDT0001 complete [ 1085.353801] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1086.446752] Key type lgssc unregistered [ 1086.574471] LNet: 67758:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.578444] LNet: Removed LNI 192.168.203.155@tcp [ 1086.862569] Key type .llcrypt unregistered [ 1086.864384] Key type ._llcrypt unregistered [ 1099.781121] alg: No test for adler32 (adler32-zlib) [ 1100.532462] Key type ._llcrypt registered [ 1100.533969] Key type .llcrypt registered [ 1100.572789] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1100.875234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1101.101350] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1101.183288] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1101.185036] LNet: Accept secure, port 988 [ 1102.767240] Key type lgssc registered [ 1103.165860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1106.063688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.213530] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1107.227396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.620848] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1111.664112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.804894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1113.263574] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1115.845768] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1117.787751] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1119.953784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1120.065444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.414309] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1122.150893] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:321 [ 1123.850912] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1130.959596] Lustre: Failing over lustre-MDT0000 [ 1131.026504] Lustre: server umount lustre-MDT0000 complete [ 1131.489960] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1131.491898] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1131.494087] Lustre: Skipped 2 previous similar messages [ 1131.499214] LustreError: Skipped 2 previous similar messages [ 1134.006452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.040674] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.041838] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1134.042284] LustreError: 11-0: MGC192.168.203.155@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1134.044121] LustreError: Skipped 2 previous similar messages [ 1134.047729] Lustre: Evicted from MGS (at 192.168.203.155@tcp) after server handle changed from 0x871876332cc0d278 to 0x871876332cc0d653 [ 1134.056039] Lustre: MGC192.168.203.155@tcp: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 1134.096582] Lustre: *** cfs_fail_loc=135, val=0*** [ 1134.132987] BUG: spinlock bad magic on CPU#0, llog_process_th/72661 [ 1134.134327] lock: 0xffff99d960e64cc0, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 1134.135743] CPU: 0 PID: 72661 Comm: llog_process_th Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1134.138831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1134.140943] Call Trace: [ 1134.141444] ? dump_stack+0xbb/0x10e [ 1134.142478] ? spin_dump.cold.3+0x32/0x65 [ 1134.143746] ? do_raw_spin_lock+0xce/0x190 [ 1134.144606] ? _raw_spin_lock+0x21/0x30 [ 1134.145296] ? mdd_changelog_fini+0x45/0x3a0 [mdd] [ 1134.146831] ? mdd_process_config+0x368/0x9c0 [mdd] [ 1134.148223] ? mdt_stack_fini+0x5de/0x1200 [mdt] [ 1134.148975] ? mdt_init0+0x650/0x1750 [mdt] [ 1134.150110] ? mdt_device_alloc+0x118/0x170 [mdt] [ 1134.151726] ? obd_setup+0x196/0x3e0 [obdclass] [ 1134.152762] ? class_setup+0x6ee/0x9e0 [obdclass] [ 1134.153827] ? class_process_config+0x17c7/0x2b90 [obdclass] [ 1134.154838] ? class_config_llog_handler+0xb73/0x1ab0 [obdclass] [ 1134.155841] ? llog_process_thread+0x109e/0x22a0 [obdclass] [ 1134.157182] ? keys_fill+0x10d/0x1b0 [obdclass] [ 1134.158814] ? lu_context_init+0xc8/0x240 [obdclass] [ 1134.159967] ? llog_process+0x40/0x40 [obdclass] [ 1134.161193] ? llog_process_thread_daemonize+0xe1/0x150 [obdclass] [ 1134.163236] ? kthread+0x1d1/0x200 [ 1134.163828] ? set_kthread_struct+0x70/0x70 [ 1134.164787] ? ret_from_fork+0x1f/0x30 [ 1134.167496] LustreError: 72661:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 1134.169674] LustreError: 72661:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.155@tcp: cfg command failed: rc = -2 [ 1134.172452] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1134.172452] [ 1134.175839] LustreError: 15c-8: MGC192.168.203.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1134.179662] LustreError: 72649:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1134.181695] LustreError: 72649:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 1134.183934] LustreError: 72649:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 1134.192145] Lustre: server umount lustre-MDT0000 complete [ 1134.193892] LustreError: 72649:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1136.239030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1136.273783] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1136.274244] LustreError: 11-0: MGC192.168.203.155@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1136.275247] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1136.276918] LustreError: Skipped 3 previous similar messages [ 1136.279819] Lustre: Evicted from MGS (at 192.168.203.155@tcp) after server handle changed from 0x871876332cc0d653 to 0x871876332cc0d6d1 [ 1136.286309] Lustre: MGC192.168.203.155@tcp: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 1136.364927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1136.398948] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1137.859848] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1140.611945] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1141.735357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1141.741479] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1141.758785] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:353 [ 1146.849790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.850036] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1146.854699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.857353] LustreError: Skipped 1 previous similar message [ 1148.825099] Lustre: server umount lustre-OST0000 complete [ 1151.968733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.968744] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1151.969488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1151.969495] Lustre: Skipped 1 previous similar message [ 1151.975437] Lustre: Skipped 2 previous similar messages [ 1156.306843] Lustre: server umount lustre-MDT0000 complete [ 1157.088094] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.093941] LustreError: Skipped 1 previous similar message [ 1157.644296] LustreError: 69486:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843776 with bad export cookie 9734660556727572177 [ 1157.647891] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1157.650908] LustreError: 69486:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1162.609817] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1163.680661] Key type lgssc unregistered [ 1163.807841] LNet: 74540:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.812619] LNet: Removed LNI 192.168.203.155@tcp [ 1164.125953] Key type .llcrypt unregistered [ 1164.127132] Key type ._llcrypt unregistered [ 1173.045681] alg: No test for adler32 (adler32-zlib) [ 1173.797421] Key type ._llcrypt registered [ 1173.798835] Key type .llcrypt registered [ 1173.831826] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1174.133926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1174.348582] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1174.417682] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1174.418927] LNet: Accept secure, port 988 [ 1175.999100] Key type lgssc registered [ 1176.352423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1179.168162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.341032] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.356348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.841125] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1185.010790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.155125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1186.720929] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1189.285834] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1191.180904] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1193.394017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1193.494792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1194.898045] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1196.581115] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:385 [ 1197.324754] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1204.246247] Lustre: Failing over lustre-MDT0000 [ 1204.370029] Lustre: server umount lustre-MDT0000 complete [ 1205.730824] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1205.731364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1205.734055] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.736952] Lustre: Skipped 2 previous similar messages [ 1205.744077] LustreError: Skipped 2 previous similar messages [ 1207.698985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.738911] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1207.738914] LustreError: 11-0: MGC192.168.203.155@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1207.738929] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.743168] Lustre: Evicted from MGS (at 192.168.203.155@tcp) after server handle changed from 0x23fe9e635dbeb99e to 0x23fe9e635dbebd79 [ 1207.744444] LustreError: Skipped 2 previous similar messages [ 1207.748979] Lustre: MGC192.168.203.155@tcp: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 1207.791200] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1207.831416] LustreError: 79156:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-9) [ 1207.833626] LustreError: 79156:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.155@tcp: cfg command failed: rc = -9 [ 1207.836597] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1207.836597] [ 1207.839755] LustreError: 15c-8: MGC192.168.203.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1207.845033] LustreError: 79144:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1207.847946] LustreError: 79144:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -9 [ 1207.852149] LustreError: 79144:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 1207.862184] Lustre: server umount lustre-MDT0000 complete [ 1207.863614] LustreError: 79144:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1209.930858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.963958] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.965566] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1209.967318] LustreError: Skipped 1 previous similar message [ 1209.969123] LustreError: 11-0: MGC192.168.203.155@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1209.971304] Lustre: Evicted from MGS (at 192.168.203.155@tcp) after server handle changed from 0x23fe9e635dbebd79 to 0x23fe9e635dbebe0c [ 1209.976459] Lustre: MGC192.168.203.155@tcp: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 1210.060581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.086732] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1211.461915] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1213.828604] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1215.462285] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1215.469111] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1215.486166] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:417 [ 1220.577307] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1220.578267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.580662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1220.583383] Lustre: Skipped 1 previous similar message [ 1222.609593] Lustre: server umount lustre-OST0000 complete [ 1225.697225] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1225.697738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.704749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.707231] Lustre: Skipped 2 previous similar messages [ 1229.898114] Lustre: server umount lustre-MDT0000 complete [ 1230.816072] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.820007] LustreError: Skipped 1 previous similar message [ 1231.192921] LustreError: 76786:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843850 with bad export cookie 2593684585023716876 [ 1231.195237] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1231.197744] LustreError: 76786:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1236.151996] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1237.211334] Key type lgssc unregistered [ 1237.324384] LNet: 81040:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.328249] LNet: Removed LNI 192.168.203.155@tcp [ 1237.661835] Key type .llcrypt unregistered [ 1237.662991] Key type ._llcrypt unregistered [ 1246.157910] alg: No test for adler32 (adler32-zlib) [ 1246.909306] Key type ._llcrypt registered [ 1246.910271] Key type .llcrypt registered [ 1246.941768] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1247.265670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1247.453658] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1247.519949] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1247.521514] LNet: Accept secure, port 988 [ 1249.103121] Key type lgssc registered [ 1249.446892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1251.895980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.025222] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.037788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1254.402172] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1257.141803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.253538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1258.643318] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1261.142531] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1263.124118] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1265.325397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1265.433025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.813280] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1268.519903] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:449 [ 1269.309799] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1277.920880] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1277.921795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.924952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1277.927845] Lustre: Skipped 1 previous similar message [ 1277.930378] Lustre: Skipped 1 previous similar message [ 1283.038643] Lustre: server umount lustre-OST0000 complete [ 1288.161325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1288.161995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1288.167209] Lustre: Skipped 1 previous similar message [ 1288.171978] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1290.440196] Lustre: server umount lustre-MDT0000 complete [ 1291.675168] LustreError: 82478:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843910 with bad export cookie 11862618723893149265 [ 1291.678275] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1291.680082] LustreError: 82478:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1291.811465] Lustre: server umount lustre-MDT0001 complete [ 1296.464095] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1297.603306] Key type lgssc unregistered [ 1297.723357] LNet: 86153:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.725987] LNet: Removed LNI 192.168.203.155@tcp [ 1298.042585] Key type .llcrypt unregistered [ 1298.044058] Key type ._llcrypt unregistered [ 1306.606082] alg: No test for adler32 (adler32-zlib) [ 1307.358359] Key type ._llcrypt registered [ 1307.360017] Key type .llcrypt registered [ 1307.401379] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1307.757331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1308.016331] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1308.094492] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1308.096554] LNet: Accept secure, port 988 [ 1309.679132] Key type lgssc registered [ 1310.038142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1312.680111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1313.817666] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.832851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1315.241341] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1317.980492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.104424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1319.630253] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1322.182507] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1324.164973] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1326.410451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1326.509622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1327.908096] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1329.573184] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 1330.425432] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1339.872682] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1339.873979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.875694] LustreError: Skipped 1 previous similar message [ 1339.879073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1339.881273] Lustre: Skipped 1 previous similar message [ 1343.588606] Lustre: server umount lustre-OST0000 complete [ 1344.992462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1344.993179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1344.996892] Lustre: Skipped 2 previous similar messages [ 1350.111809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1350.113305] Lustre: Skipped 2 previous similar messages [ 1355.232031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.234297] Lustre: Skipped 1 previous similar message [ 1359.327118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1359.372962] Lustre: server umount lustre-MDT0000 complete [ 1360.352928] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.357723] LustreError: Skipped 1 previous similar message [ 1360.610972] LustreError: 87593:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843979 with bad export cookie 6069237529976800207 [ 1360.613238] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1360.614182] LustreError: 87593:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1360.725759] Lustre: server umount lustre-MDT0001 complete [ 1365.248740] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1366.290409] Key type lgssc unregistered [ 1366.411373] LNet: 91267:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1366.416130] LNet: Removed LNI 192.168.203.155@tcp [ 1366.736774] Key type .llcrypt unregistered [ 1366.738055] Key type ._llcrypt unregistered [ 1375.632190] alg: No test for adler32 (adler32-zlib) [ 1376.383270] Key type ._llcrypt registered [ 1376.384181] Key type .llcrypt registered [ 1376.423839] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1376.837114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1377.124084] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1377.204387] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1377.206694] LNet: Accept secure, port 988 [ 1378.791131] Key type lgssc registered [ 1379.133452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1381.637599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.774610] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.787555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1384.187996] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1386.988196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.105584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1388.531730] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1391.019392] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1392.973588] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1395.274291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1395.393460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.805687] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1398.500972] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:513 [ 1399.260744] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1408.994465] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1408.994530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1408.996769] LustreError: Skipped 1 previous similar message [ 1409.003258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1409.004732] Lustre: Skipped 1 previous similar message [ 1413.466626] Lustre: server umount lustre-OST0000 complete [ 1417.696035] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1417.698403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.701464] Lustre: Skipped 1 previous similar message [ 1417.702881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1419.232875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1419.233321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.941665] Lustre: server umount lustre-MDT0000 complete [ 1422.167983] LustreError: 92707:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844041 with bad export cookie 8811905609674889811 [ 1422.170886] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1422.172377] LustreError: 92707:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1422.292586] Lustre: server umount lustre-MDT0001 complete [ 1426.796708] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1427.854685] Key type lgssc unregistered [ 1427.971331] LNet: 96379:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.973752] LNet: Removed LNI 192.168.203.155@tcp [ 1428.263897] Key type .llcrypt unregistered [ 1428.264937] Key type ._llcrypt unregistered [ 1436.517936] alg: No test for adler32 (adler32-zlib) [ 1437.269426] Key type ._llcrypt registered [ 1437.270416] Key type .llcrypt registered [ 1437.311517] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1437.667164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1437.873233] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1437.946128] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1437.947678] LNet: Accept secure, port 988 [ 1439.527120] Key type lgssc registered [ 1439.909152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1442.559798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1459.039311] LustreError: 97813:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 1: rc = -5 [ 1490.783225] LustreError: 97813:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 1520.479215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.896365] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1524.305272] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1526.195416] Lustre: server umount lustre-OST0000 complete [ 1528.933239] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:35:47 (1759844147) [ 1531.160194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.311924] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.322538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1532.825712] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1535.729710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.872417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1537.509208] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1539.949066] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1541.879956] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1544.201537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1544.329143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.846703] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1546.409592] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:545 [ 1548.433737] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1556.959924] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1556.962037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1556.965196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1561.056661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1561.057558] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1561.061020] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1563.043224] Lustre: server umount lustre-OST0000 complete [ 1566.176515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.177122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.179445] Lustre: Skipped 1 previous similar message [ 1566.181085] Lustre: Skipped 1 previous similar message [ 1570.376907] Lustre: server umount lustre-MDT0000 complete [ 1571.295991] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1571.301257] LustreError: Skipped 1 previous similar message [ 1571.670868] LustreError: 99391:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844190 with bad export cookie 11222307681160278576 [ 1571.673336] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1571.674028] LustreError: 99391:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1576.446729] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1577.467609] Key type lgssc unregistered [ 1577.584361] LNet: 103189:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.586465] LNet: Removed LNI 192.168.203.155@tcp [ 1577.865671] Key type .llcrypt unregistered [ 1577.866618] Key type ._llcrypt unregistered [ 1587.008597] alg: No test for adler32 (adler32-zlib) [ 1587.760356] Key type ._llcrypt registered [ 1587.761246] Key type .llcrypt registered [ 1587.800612] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1588.364176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1588.544234] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1588.624397] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1588.626726] LNet: Accept secure, port 988 [ 1590.207082] Key type lgssc registered [ 1590.529898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1593.213362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1594.358746] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1594.373928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.770577] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1598.503678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.624595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1600.111953] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1602.550845] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1604.503587] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1606.735627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1606.855865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.367722] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1609.959061] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:577 [ 1610.921027] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1619.428547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.429481] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1619.429730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1619.431370] Lustre: Skipped 1 previous similar message [ 1624.557764] Lustre: server umount lustre-OST0000 complete [ 1629.664419] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1629.664551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.665108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1629.665112] Lustre: Skipped 1 previous similar message [ 1629.676882] Lustre: Skipped 1 previous similar message [ 1631.873487] Lustre: server umount lustre-MDT0000 complete [ 1633.135821] LustreError: 104631:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844252 with bad export cookie 15466264516283088350 [ 1633.139911] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1633.140632] LustreError: 104631:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1633.258677] Lustre: server umount lustre-MDT0001 complete [ 1638.150306] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1639.257657] Key type lgssc unregistered [ 1639.385360] LNet: 108302:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.387647] LNet: Removed LNI 192.168.203.155@tcp [ 1639.681986] Key type .llcrypt unregistered [ 1639.683317] Key type ._llcrypt unregistered [ 1646.677770] alg: No test for adler32 (adler32-zlib) [ 1647.429330] Key type ._llcrypt registered [ 1647.430310] Key type .llcrypt registered [ 1647.472135] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1647.920754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1648.112657] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1648.188618] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1648.190033] LNet: Accept secure, port 988 [ 1649.775119] Key type lgssc registered [ 1650.123846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1652.861188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1669.343282] LustreError: 109592:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 1: rc = -5 [ 1700.063165] LustreError: 109592:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 1730.783165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.162107] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1734.579346] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1736.826286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1753.372475] LustreError: 15c-8: MGC192.168.203.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1753.379324] LustreError: 110577:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1753.383360] LustreError: 110577:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 1753.387407] LustreError: 110577:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 1753.446607] Lustre: server umount lustre-MDT0000 complete [ 1753.447981] LustreError: 110577:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1761.247149] Lustre: 111009:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844374/real 1759844374] req@000000008f2a7839 x1845330462115776/t0(0) o251->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759844380 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1761.300795] Lustre: server umount lustre-OST0000 complete [ 1764.273858] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1767.168987] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1770.408335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1772.671213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1774.640784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1776.708899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1778.969458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1778.995187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.078965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1779.088353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1779.127639] Lustre: lustre-MDT0000: new disk, initializing [ 1779.152577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.159401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1780.665279] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1784.894717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1784.917404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.956059] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1784.974117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1784.977146] Lustre: Skipped 1 previous similar message [ 1785.019064] Lustre: lustre-MDT0001: new disk, initializing [ 1785.048317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1785.058154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1785.061417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1786.455614] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1790.260144] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1792.197699] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1794.450337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1794.474253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1794.476495] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1794.568908] Lustre: lustre-OST0000: new disk, initializing [ 1794.570559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1794.593860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1795.972231] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1799.875099] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1801.472257] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1804.275644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1804.280045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1804.615945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1806.029963] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1806.103424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1809.377548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.377707] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1809.378261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1809.381412] Lustre: Skipped 1 previous similar message [ 1812.828629] Lustre: server umount lustre-OST0000 complete [ 1814.496101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1814.499487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1814.501719] Lustre: Skipped 2 previous similar messages [ 1819.615872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1819.617601] Lustre: Skipped 1 previous similar message [ 1820.224941] Lustre: server umount lustre-MDT0000 complete [ 1821.434902] LustreError: 113451:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844440 with bad export cookie 1443863850567553919 [ 1821.437673] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1821.439370] LustreError: 113451:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1824.194353] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:40:42 (1759844442) [ 1825.317760] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1830.522301] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1833.503751] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1837.153054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1839.639634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1841.611779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1843.691492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1845.964061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1845.990698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.086646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1846.096534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1846.137762] Lustre: lustre-MDT0000: new disk, initializing [ 1846.162769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.170452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1847.600912] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1851.703378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1851.732161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.763837] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1851.779499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1851.782247] Lustre: Skipped 1 previous similar message [ 1851.818281] Lustre: lustre-MDT0001: new disk, initializing [ 1851.855905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1851.858573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1853.318460] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1857.087432] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1859.006573] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1861.241926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1861.271622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1861.273895] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1861.365311] Lustre: lustre-OST0000: new disk, initializing [ 1861.366976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1861.395725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.397637] Lustre: Skipped 1 previous similar message [ 1862.944265] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1866.693867] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1868.165495] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1870.324127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1870.328587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1871.296176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1872.669876] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1872.741710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1875.425323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.426225] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1875.430961] Lustre: Skipped 1 previous similar message [ 1875.434596] LustreError: Skipped 2 previous similar messages [ 1875.435161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1875.441576] Lustre: Skipped 1 previous similar message [ 1879.446271] Lustre: server umount lustre-OST0000 complete [ 1879.448289] Lustre: Skipped 1 previous similar message [ 1882.591831] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1882.593865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.597217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.030112] LustreError: 120595:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844506 with bad export cookie 1443863850567554759 [ 1888.033539] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.035639] LustreError: 120595:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1888.157547] Lustre: server umount lustre-MDT0001 complete [ 1888.159469] Lustre: Skipped 1 previous similar message [ 1890.965338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.123903] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1891.146738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.576748] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1895.364963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.913228] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1899.310590] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1901.119332] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1903.290509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1904.732785] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1906.979750] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1914.115736] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1915.875295] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1915.875899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.876604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1915.876612] Lustre: Skipped 2 previous similar messages [ 1915.883795] Lustre: Skipped 2 previous similar messages [ 1921.178206] Lustre: server umount lustre-OST0000 complete [ 1926.112520] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1926.112641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.117766] Lustre: Skipped 1 previous similar message [ 1929.796036] LustreError: 124958:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844548 with bad export cookie 1443863850567555599 [ 1929.799114] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1929.799956] LustreError: 124958:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1934.785091] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1935.813303] Key type lgssc unregistered [ 1935.931388] LNet: 128727:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.934328] LNet: Removed LNI 192.168.203.155@tcp [ 1936.253105] Key type .llcrypt unregistered [ 1936.254393] Key type ._llcrypt unregistered [ 1942.134491] alg: No test for adler32 (adler32-zlib) [ 1942.886279] Key type ._llcrypt registered [ 1942.887292] Key type .llcrypt registered [ 1942.923706] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1948.640264] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1948.956837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1949.202496] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1949.268311] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1949.269698] LNet: Accept secure, port 988 [ 1950.839082] Key type lgssc registered [ 1951.140348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1953.883044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1955.946626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1957.945032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1959.947414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1962.136435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1962.161258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.242942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1963.256160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1963.294348] Lustre: lustre-MDT0000: new disk, initializing [ 1963.318426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.324181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1964.738528] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1968.853072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1968.881743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.919908] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1968.939283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1968.942206] Lustre: Skipped 1 previous similar message [ 1968.982394] Lustre: lustre-MDT0001: new disk, initializing [ 1969.013813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.025414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1969.028196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1970.408882] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1974.122512] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1976.004670] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1978.198402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1978.217994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1978.220264] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1978.310143] Lustre: lustre-OST0000: new disk, initializing [ 1978.311764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1978.340291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.752254] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 1983.513419] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1984.867877] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1985.019231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1985.022391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1985.966440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1987.254060] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1987.331233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1990.113700] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1990.113758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.117380] LustreError: Skipped 1 previous similar message [ 1990.120697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1990.123607] Lustre: Skipped 1 previous similar message [ 1994.012711] Lustre: server umount lustre-OST0000 complete [ 1995.232087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.232655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1995.235543] Lustre: Skipped 2 previous similar messages [ 1995.240529] Lustre: Skipped 1 previous similar message [ 2000.352132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2000.355015] Lustre: Skipped 1 previous similar message [ 2001.281337] Lustre: server umount lustre-MDT0000 complete [ 2002.470824] LustreError: 132006:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844621 with bad export cookie 7315513066749045514 [ 2002.473653] LustreError: 132006:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2002.473750] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2002.579775] Lustre: server umount lustre-MDT0001 complete [ 2005.276467] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:43:43 (1759844623) [ 2007.293895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.447679] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.467901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.715184] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2011.361510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.712513] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2015.018199] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2016.866376] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2021.856468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.856500] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2021.856978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.859609] Lustre: Skipped 1 previous similar message [ 2023.620557] Lustre: server umount lustre-MDT0000 complete [ 2024.837427] LustreError: 137356:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844643 with bad export cookie 7315513066749046354 [ 2024.840927] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2024.843023] LustreError: 137356:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2027.826298] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:44:05 (1759844645) [ 2030.134642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2047.647658] LustreError: 139093:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 1: rc = -5 [ 2058.911351] LustreError: 139093:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 2087.583200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2087.585139] Lustre: Skipped 1 previous similar message [ 2088.883853] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2091.212903] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2091.907097] Lustre: server umount lustre-OST0000 complete [ 2091.908287] Lustre: Skipped 1 previous similar message [ 2094.563555] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:45:12 (1759844712) [ 2096.673038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2096.814362] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.836678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.145638] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2100.795130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2102.454745] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2104.971557] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2106.819100] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2109.000076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2110.498601] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2112.951863] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2121.695981] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2121.697879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.700942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2121.702495] Lustre: Skipped 1 previous similar message [ 2125.795353] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2126.856395] Lustre: server umount lustre-MDT0000 complete [ 2128.143257] LustreError: 140477:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844747 with bad export cookie 7315513066749047005 [ 2128.146183] LustreError: 140477:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2128.147679] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2135.519135] Lustre: 143547:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844748/real 1759844748] req@000000001f902ec6 x1845330777769024/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759844754 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2138.315108] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:45:56 (1759844756) [ 2140.398947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2140.567279] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2140.588759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2140.590353] Lustre: Skipped 2 previous similar messages [ 2142.057283] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2144.897234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.569829] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2149.111783] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2151.085534] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2153.303996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2154.733846] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2155.493055] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:65 [ 2157.202203] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2158.697412] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2158.774247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2160.278400] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2160.364442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2164.704554] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2164.707196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.710207] Lustre: Skipped 1 previous similar message [ 2164.711955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2164.714914] Lustre: Skipped 4 previous similar messages [ 2167.063728] Lustre: server umount lustre-OST0000 complete [ 2167.064970] Lustre: Skipped 2 previous similar messages [ 2174.943984] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.949796] LustreError: Skipped 1 previous similar message [ 2175.595028] LustreError: 144197:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844794 with bad export cookie 7315513066749048055 [ 2175.597082] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2175.598186] LustreError: 144197:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2178.525217] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:46:36 (1759844796) [ 2180.687217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2197.151473] LustreError: 148536:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 1: rc = -5 [ 2208.415370] LustreError: 148536:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 2237.087274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.089072] Lustre: Skipped 2 previous similar messages [ 2238.600599] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2240.975603] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2243.154631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.971499] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.011245] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:97 [ 2258.311048] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2261.061214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.619294] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2265.060320] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2266.943147] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2268.417313] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2268.481757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2269.865250] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2269.935590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2271.712846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.713491] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2271.713632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2271.713637] Lustre: Skipped 2 previous similar messages [ 2271.717921] Lustre: Skipped 4 previous similar messages [ 2276.677629] Lustre: server umount lustre-MDT0000 complete [ 2276.679645] Lustre: Skipped 2 previous similar messages [ 2276.832764] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.836244] LustreError: Skipped 2 previous similar messages [ 2277.896892] LustreError: 148542:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844896 with bad export cookie 7315513066749049077 [ 2277.900163] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2285.535084] Lustre: 152249:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844898/real 1759844898] req@00000000c6552312 x1845330777789952/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759844904 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2288.325545] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:48:26 (1759844906) [ 2290.393948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2307.871372] LustreError: 152892:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 1: rc = -5 [ 2318.111333] LustreError: 152892:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 2347.807249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.808965] Lustre: Skipped 2 previous similar messages [ 2349.062779] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2351.400411] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2353.551076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.280651] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.332552] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:129 [ 2366.699775] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2369.443188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.882762] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2373.283673] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2375.067055] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2377.208622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2377.233922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2377.236189] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2377.278366] Lustre: lustre-OST0001: new disk, initializing [ 2377.280044] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2378.552603] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2381.109739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2381.112398] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2382.278700] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2383.718825] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2383.796018] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2385.140604] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 2385.218395] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2385.887592] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2385.889610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.892781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2385.894204] Lustre: Skipped 1 previous similar message [ 2400.223088] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2400.240335] Lustre: server umount lustre-OST0000 complete [ 2400.242242] Lustre: Skipped 2 previous similar messages [ 2401.762028] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.765582] LustreError: Skipped 3 previous similar messages [ 2410.463685] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2410.465669] LustreError: Skipped 2 previous similar messages [ 2423.263199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2424.458960] LustreError: 157223:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845043 with bad export cookie 7315513066749049994 [ 2424.461641] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2424.461782] LustreError: 157223:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2426.287708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2427.466332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2428.659152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2429.816663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2431.815457] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:50:49 (1759845049) [ 2432.317320] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2432.811361] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2433.323240] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:50:51 (1759845051) [ 2435.267211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.359177] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2435.367543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2436.595497] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2439.135304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.162235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2439.170300] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2440.512505] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2442.737382] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2444.573466] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2446.653921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.715634] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2448.009041] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2450.229021] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2450.755070] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:161 [ 2451.506983] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2451.565334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2452.768595] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2452.831066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2456.032527] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2456.032842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.034595] LustreError: Skipped 1 previous similar message [ 2456.037435] Lustre: Skipped 5 previous similar messages [ 2456.038981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2456.042728] Lustre: Skipped 18 previous similar messages [ 2473.137450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2474.659828] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2477.038386] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2478.674060] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2488.549886] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2488.553093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2488.553233] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 2488.555569] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:162 to 0x280000400:193 [ 2488.559529] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:193 [ 2488.559868] Lustre: Skipped 1 previous similar message [ 2488.966514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2490.281428] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2490.350966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2492.231772] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8cb9848ec800.ost_server_uuid 40 [ 2492.762337] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8cb9848ec800.ost_server_uuid in FULL state after 0 sec [ 2492.820825] LustreError: 159470:0:(ldlm_lockd.c:721:ldlm_handle_ast_error()) ### client (nid 192.168.203.55@tcp) returned error from blocking AST (req@0000000067aa7908 x1845330777829440 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b6695809/0x6585ec5ffd4e4597 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.55@tcp remote: 0xce2209e292065819 expref: 6 pid: 160271 timeout: 2592 lvb_type: 0 [ 2492.828895] LustreError: 138-a: lustre-MDT0000: A client on nid 192.168.203.55@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2492.831321] LustreError: 159461:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.55@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b6695809/0x6585ec5ffd4e4597 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.55@tcp remote: 0xce2209e292065819 expref: 7 pid: 160271 timeout: 0 lvb_type: 0 [ 2508.255111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2516.765285] LustreError: 159456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845135 with bad export cookie 7315513066749051142 [ 2516.767521] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2516.769147] LustreError: 159456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2531.295094] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2531.378427] Lustre: server umount lustre-MDT0001 complete [ 2531.380536] Lustre: Skipped 6 previous similar messages [ 2535.784039] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2536.746705] Key type lgssc unregistered [ 2536.847411] LNet: 165927:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2536.849743] LNet: Removed LNI 192.168.203.155@tcp [ 2537.125604] Key type .llcrypt unregistered [ 2537.127432] Key type ._llcrypt unregistered [ 2545.464780] alg: No test for adler32 (adler32-zlib) [ 2546.216263] Key type ._llcrypt registered [ 2546.217110] Key type .llcrypt registered [ 2546.251924] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2546.775479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2546.930518] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2547.006608] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2547.008062] LNet: Accept secure, port 988 [ 2548.583118] Key type lgssc registered [ 2548.913739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2551.439545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.587820] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.602298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.971176] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2556.719610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.834337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2558.190149] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2560.502266] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2562.136139] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2564.207392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2564.319779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.623277] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2567.397500] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 2567.713844] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2574.754904] Lustre: Failing over lustre-MDT0000 [ 2574.860716] Lustre: server umount lustre-MDT0000 complete [ 2576.865116] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.865609] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.867984] Lustre: Skipped 1 previous similar message [ 2576.871804] LustreError: Skipped 2 previous similar messages [ 2581.984132] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.987903] LustreError: Skipped 2 previous similar messages [ 2584.031085] Lustre: 166701:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759845195/real 1759845195] req@000000001eccf067 x1845331403744448/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759845202 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2584.038910] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2584.042773] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.046306] LustreError: Skipped 2 previous similar messages [ 2587.688488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.151926] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.154108] Lustre: Evicted from MGS (at 192.168.203.155@tcp) after server handle changed from 0xfcd8943036b25f25 to 0xfcd8943036b26300 [ 2589.154181] LustreError: Skipped 2 previous similar messages [ 2589.157279] Lustre: MGC192.168.203.155@tcp: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 2589.227909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.258641] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2590.485600] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2591.124933] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2591.126788] Lustre: lustre-MDT0000: Denying connection for new client 99a28dd6-8c86-4395-8987-1b23a1e86eb4 (at 192.168.203.55@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2594.278314] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2599.393290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2599.394313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2599.396500] Lustre: Skipped 2 previous similar messages [ 2599.399098] Lustre: Skipped 1 previous similar message [ 2604.512053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.514198] Lustre: Skipped 1 previous similar message [ 2609.632116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2612.191090] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2612.252616] Lustre: server umount lustre-OST0000 complete [ 2613.461032] LustreError: 171238:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2613.463508] Lustre: 170537:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2613.465596] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 2613.478632] Lustre: 170537:0:(mdt_handler.c:7532:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 [ 2614.752739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.756069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2614.759509] Lustre: Skipped 1 previous similar message [ 2619.593769] Lustre: server umount lustre-MDT0000 complete [ 2619.872211] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.787893] LustreError: 167368:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845239 with bad export cookie 18219475227324343040 [ 2620.790236] LustreError: 166698:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2620.790248] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2620.790803] LustreError: 167368:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2635.231110] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2635.324751] Lustre: server umount lustre-MDT0001 complete [ 2639.775173] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2640.613502] Key type lgssc unregistered [ 2640.711334] LNet: 172044:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2640.713506] LNet: Removed LNI 192.168.203.155@tcp [ 2640.952780] Key type .llcrypt unregistered [ 2640.953814] Key type ._llcrypt unregistered [ 2649.376983] alg: No test for adler32 (adler32-zlib) [ 2650.128369] Key type ._llcrypt registered [ 2650.129225] Key type .llcrypt registered [ 2650.170864] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2650.641263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2650.854462] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2650.927139] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2650.928759] LNet: Accept secure, port 988 [ 2652.503086] Key type lgssc registered [ 2652.819379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2655.236617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.371183] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.385814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.748412] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2660.489341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.592525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2661.865126] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2664.136985] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2665.914239] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2667.973552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2668.073148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2669.312525] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2671.140040] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 2671.566285] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2675.233226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2675.233387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.760025] Lustre: server umount lustre-OST0000 complete [ 2681.311660] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2681.313631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.316718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2681.318328] Lustre: Skipped 1 previous similar message [ 2685.407857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2685.408320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.527872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.529529] Lustre: Skipped 2 previous similar messages [ 2695.135138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2695.173771] Lustre: server umount lustre-MDT0000 complete [ 2695.649035] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.652406] LustreError: Skipped 1 previous similar message [ 2696.315119] LustreError: 173501:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845315 with bad export cookie 3538013269334237562 [ 2696.317662] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2696.318187] LustreError: 173501:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2696.431177] Lustre: server umount lustre-MDT0001 complete [ 2700.669585] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2701.566959] Key type lgssc unregistered [ 2701.686292] LNet: 177172:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2701.689182] LNet: Removed LNI 192.168.203.155@tcp [ 2701.940256] Key type .llcrypt unregistered [ 2701.941199] Key type ._llcrypt unregistered [ 2713.016447] alg: No test for adler32 (adler32-zlib) [ 2713.767342] Key type ._llcrypt registered [ 2713.768331] Key type .llcrypt registered [ 2713.804030] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2714.183538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2714.341791] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2714.408933] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2714.410939] LNet: Accept secure, port 988 [ 2715.983117] Key type lgssc registered [ 2716.276961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2718.602120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.732330] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2719.745089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.038422] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2723.590134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.698239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2725.014565] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2727.359061] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2729.183886] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2731.280259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2731.381958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.652512] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2734.436657] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 2734.845448] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2743.776865] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2743.777286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.782898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2743.784374] Lustre: Skipped 1 previous similar message [ 2748.177686] Lustre: server umount lustre-OST0000 complete [ 2749.407587] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2749.409693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.412570] Lustre: Skipped 1 previous similar message [ 2749.414113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.016345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.016915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.528429] Lustre: server umount lustre-MDT0000 complete [ 2756.744086] LustreError: 178901:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845375 with bad export cookie 2650806944546661350 [ 2756.747294] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2756.749666] LustreError: 178901:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2756.876313] Lustre: server umount lustre-MDT0001 complete [ 2761.377848] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2762.335034] Key type lgssc unregistered [ 2762.448305] LNet: 182575:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2762.450881] LNet: Removed LNI 192.168.203.155@tcp [ 2762.727599] Key type .llcrypt unregistered [ 2762.728596] Key type ._llcrypt unregistered [ 2770.371182] alg: No test for adler32 (adler32-zlib) [ 2771.122401] Key type ._llcrypt registered [ 2771.123272] Key type .llcrypt registered [ 2771.157695] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2771.541993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2771.783929] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2771.858487] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2771.860070] LNet: Accept secure, port 988 [ 2773.439068] Key type lgssc registered [ 2773.795841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2777.149489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.300281] Lustre: *** cfs_fail_loc=135, val=0*** [ 2778.346475] LustreError: 184144:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 2778.348630] LustreError: 184144:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.155@tcp: cfg command failed: rc = -2 [ 2778.351469] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2778.351469] [ 2778.354100] LustreError: 15c-8: MGC192.168.203.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2778.357599] LustreError: 184105:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2778.360963] LustreError: 184105:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 2778.363035] LustreError: 184105:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 2784.735212] Lustre: 184105:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759845397/real 1759845397] req@00000000b97b97a9 x1845331639665472/t0(0) o251->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759845403 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2784.801591] Lustre: server umount lustre-MDT0000 complete [ 2784.802846] LustreError: 184105:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2786.877429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2787.028391] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.042501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2788.421916] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2791.035558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.152933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2792.450565] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2794.733648] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2796.461877] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2801.633900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2801.633914] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2801.634510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2801.637447] Lustre: Skipped 1 previous similar message [ 2803.267440] Lustre: server umount lustre-MDT0000 complete [ 2804.547084] LustreError: 184497:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845423 with bad export cookie 4527562276453303223 [ 2804.550445] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2804.551601] LustreError: 184497:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2804.687328] Lustre: server umount lustre-MDT0001 complete [ 2808.946421] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2810.472643] Key type lgssc unregistered [ 2810.627351] LNet: 186999:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2810.631064] LNet: Removed LNI 192.168.203.155@tcp [ 2811.094639] Key type .llcrypt unregistered [ 2811.095985] Key type ._llcrypt unregistered [ 2822.711021] alg: No test for adler32 (adler32-zlib) [ 2823.461363] Key type ._llcrypt registered [ 2823.462756] Key type .llcrypt registered [ 2823.524524] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2823.661181] Key type .llcrypt unregistered [ 2823.662711] Key type ._llcrypt unregistered [ 2832.642044] alg: No test for adler32 (adler32-zlib) [ 2833.394349] Key type ._llcrypt registered [ 2833.396594] Key type .llcrypt registered [ 2833.474619] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2834.102759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2834.289699] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2834.387805] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2834.390084] LNet: Accept secure, port 988 [ 2835.991147] Key type lgssc registered [ 2836.321621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2838.799608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2855.263387] LustreError: 189251:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 1: rc = -5 [ 2887.017675] LustreError: 189251:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 2917.791224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2919.973378] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2923.960466] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2927.536407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.493855] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.539732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.586187] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:321 [ 2941.697427] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2944.993475] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.663467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.828419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2947.686404] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 2951.280197] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2953.899513] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2955.264862] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 2960.354983] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2960.358618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.365973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.381327] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2961.383558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.384312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2964.215865] Lustre: server umount lustre-OST0000 complete [ 2966.497715] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2966.500835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2966.506476] Lustre: Skipped 1 previous similar message [ 2966.512071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2966.518659] Lustre: Skipped 2 previous similar messages [ 2971.616686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2971.619450] Lustre: Skipped 1 previous similar message [ 2972.562856] Lustre: server umount lustre-MDT0000 complete [ 2974.664769] LustreError: 189258:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845593 with bad export cookie 11016918099015889857 [ 2974.666558] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2974.670653] LustreError: 189258:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2974.837585] Lustre: server umount lustre-MDT0001 complete [ 2979.663603] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2980.951223] Key type lgssc unregistered [ 2981.104554] LNet: 193191:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2981.114850] LNet: Removed LNI 192.168.203.155@tcp [ 2981.472633] Key type .llcrypt unregistered [ 2981.474211] Key type ._llcrypt unregistered [ 2992.071967] alg: No test for adler32 (adler32-zlib) [ 2992.824473] Key type ._llcrypt registered [ 2992.826659] Key type .llcrypt registered [ 2992.891968] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2993.365419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2993.589819] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2993.687670] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2993.690443] LNet: Accept secure, port 988 [ 2995.287181] Key type lgssc registered [ 2995.747788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2999.002546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.187710] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.213271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3001.996643] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3005.340254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.408790] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.495220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3007.110604] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3009.959632] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3012.216577] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3015.106253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3015.253811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3016.293171] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:353 [ 3017.185634] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3021.371093] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3023.935888] Lustre: Failing over lustre-MDT0000 [ 3024.020532] Lustre: server umount lustre-MDT0000 complete [ 3025.889661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.890091] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3025.892256] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.895510] Lustre: Skipped 1 previous similar message [ 3031.009219] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3031.014163] LustreError: Skipped 5 previous similar messages [ 3033.055208] Lustre: 193962:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759845644/real 1759845644] req@00000000f383b0b4 x1845331872457536/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759845651 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3033.066538] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3038.176105] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.185563] LustreError: Skipped 7 previous similar messages [ 3038.450590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.279037] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.291390] LustreError: Skipped 5 previous similar messages [ 3049.441808] Lustre: Evicted from MGS (at 192.168.203.155@tcp) after server handle changed from 0x4a96d16e5f0acfc5 to 0x4a96d16e5f0ad3ae [ 3049.448285] Lustre: MGC192.168.203.155@tcp: Connection restored to (at 0@lo) [ 3049.548865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3049.576733] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3051.301139] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3051.395780] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3053.067920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire in log lustre-MDT0000 [ 3054.566467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3054.573549] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3054.589858] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:385 [ 3059.334636] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight in log lustre-client [ 3064.800691] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3064.801886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.804649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3064.809959] Lustre: Skipped 1 previous similar message [ 3069.923401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3069.926340] Lustre: Skipped 2 previous similar messages [ 3070.879399] Lustre: server umount lustre-OST0000 complete [ 3075.041883] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3075.042237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3075.047061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3075.055297] Lustre: Skipped 1 previous similar message [ 3078.541717] Lustre: server umount lustre-MDT0000 complete [ 3080.062375] LustreError: 194631:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845698 with bad export cookie 5374713477284418478 [ 3080.064649] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3080.065965] LustreError: 194631:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3085.651230] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3086.838385] Key type lgssc unregistered [ 3086.964637] LNet: 199814:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3086.968595] LNet: Removed LNI 192.168.203.155@tcp [ 3087.297987] Key type .llcrypt unregistered [ 3087.299809] Key type ._llcrypt unregistered [ 3097.580864] alg: No test for adler32 (adler32-zlib) [ 3098.333415] Key type ._llcrypt registered [ 3098.335030] Key type .llcrypt registered [ 3098.388591] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3098.866959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3099.034417] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3099.117147] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3099.119135] LNet: Accept secure, port 988 [ 3100.703225] Key type lgssc registered [ 3101.061226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3103.951644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.115191] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3105.129749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.700777] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3109.655819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.791892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3111.304840] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3113.921975] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3115.930225] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3118.201798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3118.312136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.719873] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3120.359066] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:417 [ 3122.138163] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3123.698791] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3131.298311] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3142.571335] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3150.304665] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3150.305883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.307939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3150.310817] Lustre: Skipped 1 previous similar message [ 3154.763380] Lustre: server umount lustre-OST0000 complete [ 3155.936064] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3155.938573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.942780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3155.944488] Lustre: Skipped 1 previous similar message [ 3160.544639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3160.544872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.663903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3165.666776] Lustre: Skipped 2 previous similar messages [ 3170.271100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3170.314378] Lustre: server umount lustre-MDT0000 complete [ 3171.595059] LustreError: 203936:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845790 with bad export cookie 16563735319048266118 [ 3171.597476] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3171.598016] LustreError: 203936:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3171.744756] Lustre: server umount lustre-MDT0001 complete [ 3176.464615] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3177.468884] Key type lgssc unregistered [ 3177.585378] LNet: 205122:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3177.589402] LNet: Removed LNI 192.168.203.155@tcp [ 3177.874202] Key type .llcrypt unregistered [ 3177.875185] Key type ._llcrypt unregistered [ 3186.653217] alg: No test for adler32 (adler32-zlib) [ 3187.404352] Key type ._llcrypt registered [ 3187.405426] Key type .llcrypt registered [ 3187.440597] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3187.723618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3187.942170] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3188.009866] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3188.011361] LNet: Accept secure, port 988 [ 3189.583077] Key type lgssc registered [ 3189.948482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3192.555947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.692950] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3193.706819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.050952] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3197.792557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.914207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3199.386264] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3201.813648] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3203.616554] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3205.747385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.880935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3207.252874] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3208.933418] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:449 [ 3209.707855] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3212.118644] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 3214.543200] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 3223.873717] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 3234.159511] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 3244.001944] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3244.003198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.006555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3244.012359] Lustre: Skipped 1 previous similar message [ 3244.017897] Lustre: Skipped 1 previous similar message [ 3246.855479] Lustre: server umount lustre-OST0000 complete [ 3249.123409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.123625] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3249.124344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3249.130249] Lustre: Skipped 1 previous similar message [ 3254.241215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.245758] Lustre: Skipped 2 previous similar messages [ 3254.295933] Lustre: server umount lustre-MDT0000 complete [ 3255.665860] LustreError: 206561:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845874 with bad export cookie 8775846436331810836 [ 3255.668593] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3255.670073] LustreError: 206561:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3255.792660] Lustre: server umount lustre-MDT0001 complete [ 3260.666816] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3261.645747] Key type lgssc unregistered [ 3261.754392] LNet: 212011:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3261.758354] LNet: Removed LNI 192.168.203.155@tcp [ 3262.127475] Key type .llcrypt unregistered [ 3262.129263] Key type ._llcrypt unregistered [ 3271.362916] alg: No test for adler32 (adler32-zlib) [ 3272.115552] Key type ._llcrypt registered [ 3272.116439] Key type .llcrypt registered [ 3272.156574] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3272.545702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3272.701341] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3272.770784] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3272.772058] LNet: Accept secure, port 988 [ 3274.343099] Key type lgssc registered [ 3274.709085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3278.250112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3279.459822] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.479983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.949065] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3283.929659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3284.078062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3285.679214] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3288.445647] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3290.590898] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3293.433044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3293.645700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3294.693716] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:481 [ 3295.022490] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3297.664463] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3301.761193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3301.802525] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3301.888257] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3303.356712] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3306.121345] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3308.778151] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3319.356456] Lustre: Permanently deactivating lustre-OST0001 [ 3319.363547] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 3326.972830] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3341.533229] Lustre: Permanently reactivating lustre-OST0001 [ 3341.538718] Lustre: Modifying parameter lustre-OST0001-osc.osc.active in log lustre-client [ 3341.541120] Lustre: Skipped 2 previous similar messages [ 3351.023457] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.027643] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3356.129169] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3356.130877] Lustre: Skipped 2 previous similar messages [ 3356.911484] Lustre: server umount lustre-OST0001 complete [ 3361.249356] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3361.249437] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3361.250623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.250633] Lustre: Skipped 1 previous similar message [ 3361.251979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3361.253359] LustreError: Skipped 1 previous similar message [ 3361.255812] LustreError: Skipped 1 previous similar message [ 3364.396210] Lustre: server umount lustre-OST0000 complete [ 3366.369390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.372831] Lustre: Skipped 1 previous similar message [ 3366.374853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3366.376996] Lustre: Skipped 2 previous similar messages [ 3371.489237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3371.492858] Lustre: Skipped 1 previous similar message [ 3380.191180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3380.240528] Lustre: server umount lustre-MDT0000 complete [ 3381.540056] LustreError: 213452:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846000 with bad export cookie 18228990478987137288 [ 3381.542647] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3381.545716] LustreError: 213452:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3386.505077] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3387.395409] Key type lgssc unregistered [ 3387.509387] LNet: 218618:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3387.512056] LNet: Removed LNI 192.168.203.155@tcp [ 3387.815022] Key type .llcrypt unregistered [ 3387.816230] Key type ._llcrypt unregistered [ 3399.244325] alg: No test for adler32 (adler32-zlib) [ 3400.006627] Key type ._llcrypt registered [ 3400.007487] Key type .llcrypt registered [ 3400.058840] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3405.533228] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3405.936056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3406.214667] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3406.287648] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3406.289617] LNet: Accept secure, port 988 [ 3407.871135] Key type lgssc registered [ 3408.229995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3411.367786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3413.771243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3415.843790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3417.993708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3420.151098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3420.192918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.284672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3421.294662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3421.335190] Lustre: lustre-MDT0000: new disk, initializing [ 3421.360331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.368854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3422.759849] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3427.085204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3427.120046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.158418] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3427.195772] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3427.196745] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3427.198178] Lustre: Skipped 1 previous similar message [ 3427.237354] Lustre: lustre-MDT0001: new disk, initializing [ 3427.270675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3427.282835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3427.285644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3428.880636] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3432.774451] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3434.853661] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3437.427163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3437.465666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3437.469104] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3437.595992] Lustre: lustre-OST0000: new disk, initializing [ 3437.598451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3437.629672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3439.111842] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3443.039197] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3443.702770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3443.705400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3444.477543] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3444.549416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3445.993878] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3446.070479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3448.801364] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3448.801981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.803125] LustreError: Skipped 1 previous similar message [ 3448.806235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3448.808138] Lustre: Skipped 1 previous similar message [ 3452.851882] Lustre: server umount lustre-OST0000 complete [ 3458.015761] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3458.017840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.020963] Lustre: Skipped 1 previous similar message [ 3458.022338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3459.040580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.357512] Lustre: server umount lustre-MDT0000 complete [ 3461.567569] LustreError: 223556:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846080 with bad export cookie 11244010887697644220 [ 3461.569961] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3461.570569] LustreError: 223556:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3461.686495] Lustre: server umount lustre-MDT0001 complete [ 3464.153393] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 10:08:02 (1759846082) [ 3466.238940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3466.385594] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3466.406767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3467.775314] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3470.359103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.467947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3471.775452] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3474.115207] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3475.812754] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3477.808833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3479.156695] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3481.365579] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3487.985106] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3498.742217] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3504.215916] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3518.237718] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3518.241385] Lustre: Skipped 1 previous similar message [ 3536.845970] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3536.849906] Lustre: Skipped 1 previous similar message [ 3573.519520] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3573.521694] Lustre: Skipped 3 previous similar messages [ 3639.905279] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3639.907310] Lustre: Skipped 7 previous similar messages [ 3666.400114] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3666.403693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.409314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3666.412046] Lustre: Skipped 2 previous similar messages [ 3670.497486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3670.498218] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3670.499106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3671.664588] Lustre: server umount lustre-OST0000 complete [ 3675.617098] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3680.735877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3680.737582] Lustre: Skipped 4 previous similar messages [ 3687.391176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3687.451362] Lustre: server umount lustre-MDT0000 complete [ 3689.010419] LustreError: 227277:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846307 with bad export cookie 11244010887697645074 [ 3689.013795] LustreError: 227277:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3689.014902] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3694.687587] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3696.002009] Key type lgssc unregistered [ 3696.143379] LNet: 232176:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3696.146614] LNet: Removed LNI 192.168.203.155@tcp [ 3696.429235] Key type .llcrypt unregistered [ 3696.430292] Key type ._llcrypt unregistered [ 3705.523174] alg: No test for adler32 (adler32-zlib) [ 3706.274359] Key type ._llcrypt registered [ 3706.276210] Key type .llcrypt registered [ 3706.314732] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3706.828498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3707.022907] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3707.099921] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3707.101348] LNet: Accept secure, port 988 [ 3708.679080] Key type lgssc registered [ 3709.068610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3711.927618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3713.145291] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3713.174107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3715.018736] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3718.260839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.386330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3719.929467] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3723.496875] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3726.090589] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3729.063259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3729.229173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3730.851906] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all 8 [ 3733.656564] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3753.955825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.956253] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3753.966661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.971986] LustreError: Skipped 1 previous similar message [ 3753.974843] Lustre: Skipped 1 previous similar message [ 3755.462609] Lustre: server umount lustre-OST0000 complete [ 3759.073941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.075745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3759.082168] Lustre: Skipped 2 previous similar messages [ 3759.088067] Lustre: Skipped 1 previous similar message [ 3764.192784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3764.194430] Lustre: Skipped 1 previous similar message [ 3769.313206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3769.315857] Lustre: Skipped 1 previous similar message [ 3771.359097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3771.407783] Lustre: server umount lustre-MDT0000 complete [ 3772.591288] LustreError: 233616:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846391 with bad export cookie 17034645212919347053 [ 3772.594039] LustreError: 233616:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3772.594278] LustreError: 166-1: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3772.742521] Lustre: server umount lustre-MDT0001 complete [ 3777.486189] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3778.505199] Key type lgssc unregistered [ 3778.624408] LNet: 237474:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3778.626676] LNet: Removed LNI 192.168.203.155@tcp [ 3778.965529] Key type .llcrypt unregistered [ 3778.966634] Key type ._llcrypt unregistered [ 3786.918780] alg: No test for adler32 (adler32-zlib) [ 3787.669295] Key type ._llcrypt registered [ 3787.670191] Key type .llcrypt registered [ 3787.702471] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3787.779894] Key type .llcrypt unregistered [ 3787.780803] Key type ._llcrypt unregistered [ 3795.364743] alg: No test for adler32 (adler32-zlib) [ 3796.115285] Key type ._llcrypt registered [ 3796.116156] Key type .llcrypt registered [ 3796.150518] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3801.702294] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3802.079110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3802.222562] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3802.293968] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3802.295462] LNet: Accept secure, port 988 [ 3803.871113] Key type lgssc registered [ 3804.313981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3807.822063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3810.830078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3812.847260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3815.241665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3815.884467] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3694 sec ========== 10:13:53 (1759846433) [ 3831.667539] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3832.820797] Key type lgssc unregistered [ 3832.959502] LNet: 243694:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3832.964479] LNet: Removed LNI 192.168.203.155@tcp [ 3833.404403] Key type .llcrypt unregistered [ 3833.405885] Key type ._llcrypt unregistered