[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 584709275 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003161] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009044] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010027] pid_max: default: 32768 minimum: 301 [ 0.012306] LSM: Security Framework initializing [ 0.013052] Yama: becoming mindful. [ 0.014044] SELinux: Initializing. [ 0.015073] *** VALIDATE selinux *** [ 0.023245] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027803] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] *** VALIDATE tmpfs *** [ 0.031480] *** VALIDATE proc *** [ 0.032260] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035290] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036176] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037020] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038035] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042667] debug: unmapping init [mem 0xffffffffb7059000-0xffffffffb7060fff] [ 0.044203] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046263] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047032] ... version: 2 [ 0.048010] ... bit width: 48 [ 0.049019] ... generic registers: 4 [ 0.050018] ... value mask: 0000ffffffffffff [ 0.051023] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054598] rcu: Hierarchical SRCU implementation. [ 0.057261] smp: Bringing up secondary CPUs ... [ 0.058870] x86: Booting SMP configuration: [ 0.059095] .... node #0, CPUs: #1 #2 #3 [ 0.072530] smp: Brought up 1 node, 4 CPUs [ 0.074013] smpboot: Max logical packages: 1 [ 0.075012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.116089] node 0 deferred pages initialised in 36ms [ 0.122012] devtmpfs: initialized [ 0.123608] x86/mm: Memory block size: 128MB [ 0.130518] gcov: version magic: 0x41383552 [ 0.135174] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136348] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137338] pinctrl core: initialized pinctrl subsystem [ 0.138870] [ 0.139270] ************************************************************* [ 0.140032] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.141088] ** ** [ 0.142011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.143014] ** ** [ 0.144040] ** This means that this kernel is built to expose internal ** [ 0.145012] ** IOMMU data structures, which may compromise security on ** [ 0.146021] ** your system. ** [ 0.147010] ** ** [ 0.148011] ** If you see this message and you are not debugging the ** [ 0.149013] ** kernel, report this immediately to your vendor! ** [ 0.150013] ** ** [ 0.151015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152013] ************************************************************* [ 0.153987] NET: Registered protocol family 16 [ 0.154559] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.155062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.156065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.158090] cpuidle: using governor menu [ 0.159816] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.162496] PCI: Using configuration type 1 for base access [ 0.165117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.176000] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.178112] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184071] cryptd: max_cpu_qlen set to 1000 [ 0.188933] ACPI: Added _OSI(Module Device) [ 0.191017] ACPI: Added _OSI(Processor Device) [ 0.193012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.198050] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209048] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.223763] ACPI: Interpreter enabled [ 0.224170] ACPI: PM: (supports S0 S3 S4 S5) [ 0.225023] ACPI: Using IOAPIC for interrupt routing [ 0.226149] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227584] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239401] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.240067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.241019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.242075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.244276] acpiphp: Slot [2] registered [ 0.245120] acpiphp: Slot [3] registered [ 0.246098] acpiphp: Slot [4] registered [ 0.247140] acpiphp: Slot [5] registered [ 0.248085] acpiphp: Slot [6] registered [ 0.249090] acpiphp: Slot [7] registered [ 0.250065] acpiphp: Slot [8] registered [ 0.251045] acpiphp: Slot [9] registered [ 0.252121] acpiphp: Slot [10] registered [ 0.253103] acpiphp: Slot [11] registered [ 0.254091] acpiphp: Slot [12] registered [ 0.255085] acpiphp: Slot [13] registered [ 0.256091] acpiphp: Slot [14] registered [ 0.257080] acpiphp: Slot [15] registered [ 0.258083] acpiphp: Slot [16] registered [ 0.259175] acpiphp: Slot [17] registered [ 0.260142] acpiphp: Slot [18] registered [ 0.261098] acpiphp: Slot [19] registered [ 0.262089] acpiphp: Slot [20] registered [ 0.263097] acpiphp: Slot [21] registered [ 0.264095] acpiphp: Slot [22] registered [ 0.265104] acpiphp: Slot [23] registered [ 0.266085] acpiphp: Slot [24] registered [ 0.267090] acpiphp: Slot [25] registered [ 0.268068] acpiphp: Slot [26] registered [ 0.269087] acpiphp: Slot [27] registered [ 0.270090] acpiphp: Slot [28] registered [ 0.271213] acpiphp: Slot [29] registered [ 0.272092] acpiphp: Slot [30] registered [ 0.273146] acpiphp: Slot [31] registered [ 0.274082] PCI host bridge to bus 0000:00 [ 0.275040] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.276020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.277028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.278041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.279016] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.280023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.281182] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.284512] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.287430] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.293021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.296883] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.297025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.298020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.299019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.300844] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.305180] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.307051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.309654] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.315015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.328097] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.332018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.337473] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.346018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.365027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.400016] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.413000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.418016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.425016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.440015] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.451451] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.457000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.468014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.498035] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.515899] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.526017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.537016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.555017] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.564167] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.570016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.581018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.592023] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.606012] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.617016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.627016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.653030] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.674731] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.678514] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.679402] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.680000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.680296] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.690491] iommu: Default domain type: Passthrough [ 0.693942] SCSI subsystem initialized [ 0.697615] ACPI: bus type USB registered [ 0.701141] usbcore: registered new interface driver usbfs [ 0.702086] usbcore: registered new interface driver hub [ 0.708054] usbcore: registered new device driver usb [ 0.715534] pps_core: LinuxPPS API ver. 1 registered [ 0.718010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.722105] PTP clock support registered [ 0.725154] EDAC MC: Ver: 3.0.0 [ 0.728298] PCI: Using ACPI for IRQ routing [ 0.729000] NetLabel: Initializing [ 0.729011] NetLabel: domain hash size = 128 [ 0.732024] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.737298] NetLabel: unlabeled traffic allowed by default [ 0.743107] vgaarb: loaded [ 0.748677] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.751017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.761638] clocksource: Switched to clocksource kvm-clock [ 0.962796] VFS: Disk quotas dquot_6.6.0 [ 0.964106] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.971909] *** VALIDATE ramfs *** [ 0.972946] *** VALIDATE hugetlbfs *** [ 0.974055] pnp: PnP ACPI init [ 0.977304] pnp: PnP ACPI: found 6 devices [ 1.000836] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.004579] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.007008] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.009159] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.012651] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.015650] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.021374] NET: Registered protocol family 2 [ 1.024069] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.035611] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.039316] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.046497] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.050078] TCP: Hash tables configured (established 65536 bind 65536) [ 1.053194] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.059960] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.063513] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.067235] NET: Registered protocol family 1 [ 1.071556] RPC: Registered named UNIX socket transport module. [ 1.074832] RPC: Registered udp transport module. [ 1.076635] RPC: Registered tcp transport module. [ 1.079038] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.081734] NET: Registered protocol family 44 [ 1.084045] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.086632] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.089216] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.094136] PCI: CLS 0 bytes, default 64 [ 1.096629] Unpacking initramfs... [ 3.853555] debug: unmapping init [mem 0xffff9aaffcc54000-0xffff9aaffffbffff] [ 3.862123] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.866780] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.874392] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.189471] Initialise system trusted keyrings [ 5.192966] Key type blacklist registered [ 5.194952] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.205902] zbud: loaded [ 5.210704] *** VALIDATE nfs *** [ 5.212153] *** VALIDATE nfs4 *** [ 5.213637] pstore: using deflate compression [ 5.217891] Platform Keyring initialized [ 5.536092] NET: Registered protocol family 38 [ 5.542193] Key type asymmetric registered [ 5.548891] Asymmetric key parser 'x509' registered [ 5.552860] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.563388] io scheduler mq-deadline registered [ 5.568821] io scheduler kyber registered [ 5.570160] io scheduler bfq registered [ 5.573733] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.581437] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.584304] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.592636] ACPI: Power Button [PWRF] [ 5.781831] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.984370] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.370770] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.611442] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.978114] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.005453] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.048595] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.053821] Non-volatile memory driver v1.3 [ 7.055270] Linux agpgart interface v0.103 [ 7.086168] virtio_blk virtio1: [vda] 67984 512-byte logical blocks (34.8 MB/33.2 MiB) [ 7.089157] vda: detected capacity change from 0 to 34807808 [ 7.145904] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.154067] vdb: detected capacity change from 0 to 1073741824 [ 7.196663] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.205391] vdc: detected capacity change from 0 to 2621440000 [ 7.231466] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.233830] vdd: detected capacity change from 0 to 2621440000 [ 7.273892] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.278486] vde: detected capacity change from 0 to 4294967296 [ 7.314684] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.321821] vdf: detected capacity change from 0 to 4294967296 [ 7.336430] libphy: Fixed MDIO Bus: probed [ 7.354652] usbcore: registered new interface driver usbserial_generic [ 7.356144] usbserial: USB Serial support registered for generic [ 7.362425] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.371583] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.375782] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.381503] mousedev: PS/2 mouse device common for all mice [ 7.390238] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.390428] rtc_cmos 00:05: RTC can wake from S4 [ 7.400349] rtc_cmos 00:05: registered as rtc0 [ 7.402024] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.409676] intel_pstate: CPU model not supported [ 7.416435] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.421191] hid: raw HID events driver (C) Jiri Kosina [ 7.432839] usbcore: registered new interface driver usbhid [ 7.442386] usbhid: USB HID core driver [ 7.443586] drop_monitor: Initializing network drop monitor service [ 7.464223] Initializing XFRM netlink socket [ 7.467730] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.476980] NET: Registered protocol family 10 [ 7.482822] Segment Routing with IPv6 [ 7.483954] NET: Registered protocol family 17 [ 7.492046] mpls_gso: MPLS GSO support [ 7.501759] RAS: Correctable Errors collector initialized. [ 7.503452] AVX version of gcm_enc/dec engaged. [ 7.512807] AES CTR mode by8 optimization enabled [ 7.778154] sched_clock: Marking stable (7778106470, 0)->(9077059910, -1298953440) [ 7.790918] registered taskstats version 1 [ 7.798821] Loading compiled-in X.509 certificates [ 7.803638] zswap: loaded using pool lzo/zbud [ 7.883740] Key type big_key registered [ 7.930488] Key type encrypted registered [ 7.931710] ima: No TPM chip found, activating TPM-bypass! [ 7.939432] ima: Allocated hash algorithm: sha1 [ 7.949491] ima: No architecture policies found [ 7.958823] evm: Initialising EVM extended attributes: [ 7.964500] evm: security.selinux [ 7.967201] evm: security.ima [ 7.971426] evm: security.capability [ 7.977130] evm: HMAC attrs: 0x1 [ 7.986933] rtc_cmos 00:05: setting system clock to 2025-10-24 13:08:19 UTC (1761311299) [ 8.004658] debug: unmapping init [mem 0xffffffffb8003000-0xffffffffb81fffff] [ 8.014305] debug: unmapping init [mem 0xffffffffb6d82000-0xffffffffb7058fff] [ 8.024152] Write protecting the kernel read-only data: 28672k [ 8.031517] debug: unmapping init [mem 0xffffffffb5403000-0xffffffffb55fffff] [ 8.037013] debug: unmapping init [mem 0xffffffffb5d14000-0xffffffffb5dfffff] [ 8.110632] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.125603] systemd[1]: Detected virtualization kvm. [ 8.127652] systemd[1]: Detected architecture x86-64. [ 8.131138] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.161798] systemd[1]: No hostname configured. [ 8.163616] systemd[1]: Set hostname to . [ 8.165211] random: systemd: uninitialized urandom read (16 bytes read) [ 8.167951] systemd[1]: Initializing machine ID from random generator. [ 8.310294] random: ln: uninitialized urandom read (6 bytes read) [ 8.632223] random: systemd: uninitialized urandom read (16 bytes read) [ 8.636508] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 8.651726] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 8.657957] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.709243] device-mapper: uevent: version 1.0.3 [ 10.712200] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.187294] virtio_net virtio0 ens2: renamed from eth0 [ 13.363037] random: fast init done [ 13.734847] scsi host0: ata_piix [ 13.893408] scsi host1: ata_piix [ 13.894853] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.909197] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.512032] random: crng init done [ 19.513091] random: 7 urandom warning(s) missed due to ratelimiting [ 23.044731] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 25.083536] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 27.796596] printk: systemd: 26 output lines suppressed due to ratelimiting [ 28.392697] SELinux: Disabled at runtime. [ 28.473929] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.487612] systemd[1]: Detected virtualization kvm. [ 28.491983] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.260761] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.269902] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.286689] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.302586] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.322294] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.361626] systemd[1]: Starting Journal Service... Starting Journal Service... [ 30.393139] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ 30.894266] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 32.434853] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 33.841305] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.910827] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 34.826160] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.368476] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit)[ 41.666740] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 43.011895] NFS: Registering the id_resolver key type [ 43.020158] Key type id_resolver registered [ 43.024621] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started OpenSSH server daemon. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg414-server login: [ 86.344931] libcfs: loading out-of-tree module taints kernel. [ 86.371167] alg: No test for adler32 (adler32-zlib) [ 87.125436] Key type ._llcrypt registered [ 87.127021] Key type .llcrypt registered [ 87.197402] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 97.406156] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 98.076350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.567251] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 99.061637] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 99.067187] LNet: Accept secure, port 988 [ 100.759975] Key type lgssc registered [ 101.726126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 119.415445] hrtimer: interrupt took 10434860 ns [ 121.677634] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 125.848678] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.585725] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 144.774262] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 150.817563] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 160.420355] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 168.589168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 168.645578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.851331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 169.894427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 169.965893] Lustre: lustre-MDT0000: new disk, initializing [ 170.041890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.054659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.920588] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 182.155281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.213430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.268330] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 182.291422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 182.294550] Lustre: Skipped 1 previous similar message [ 182.357991] Lustre: lustre-MDT0001: new disk, initializing [ 182.426714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 182.485100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 182.491254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 185.767239] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 195.634970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 195.710158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 195.713788] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 195.877612] Lustre: lustre-OST0000: new disk, initializing [ 195.881696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 195.943969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.539565] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 202.808296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.822773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.066447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 211.120501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.124673] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 211.245871] Lustre: lustre-OST0001: new disk, initializing [ 211.247852] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 211.293335] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.030537] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 220.700415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 220.708890] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.710770] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.952536] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 235.018754] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing check_logdir /tmp/testlogs/ [ 239.025784] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing yml_node [ 242.954607] Lustre: DEBUG MARKER: Client: 2.15.7.13 [ 245.337321] Lustre: DEBUG MARKER: MDS: 2.15.7.13 [ 248.044493] Lustre: DEBUG MARKER: OSS: 2.15.7.13 [ 249.730801] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 09:12:19 EDT 2025 [ 257.842776] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball 110 [ 259.226330] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 274.911968] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 274.917780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 274.925941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 276.959814] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 276.960533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 276.975102] Lustre: Skipped 2 previous similar messages [ 276.985480] Lustre: Skipped 3 previous similar messages [ 282.084339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 282.086786] Lustre: Skipped 3 previous similar messages [ 287.203085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.205871] Lustre: Skipped 3 previous similar messages [ 288.223225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 288.418673] Lustre: server umount lustre-MDT0000 complete [ 292.147901] LustreError: 6225:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311583 with bad export cookie 2140644609837850938 [ 292.149701] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.168056] LustreError: 6225:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 292.320764] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.325389] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.330228] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 292.344189] LustreError: Skipped 1 previous similar message [ 292.354415] Lustre: Skipped 1 previous similar message [ 297.444808] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.467974] LustreError: Skipped 1 previous similar message [ 298.465192] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.490075] LustreError: Skipped 3 previous similar messages [ 298.714819] Lustre: server umount lustre-MDT0001 complete [ 301.874418] Lustre: server umount lustre-OST0000 complete [ 305.552718] Lustre: server umount lustre-OST0001 complete [ 311.783310] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 322.022905] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 332.081612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 338.780213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 344.380433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 350.449560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 356.718623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 356.789987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 357.096513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 357.143423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 357.252920] Lustre: lustre-MDT0000: new disk, initializing [ 357.399922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 357.424567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 361.762946] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 371.364134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 371.450177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.549786] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 371.601617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 371.608126] Lustre: Skipped 1 previous similar message [ 371.702593] Lustre: lustre-MDT0001: new disk, initializing [ 371.804306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 371.837738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 371.850837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 375.899090] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 384.595946] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 389.849461] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 396.042210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 396.134802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 396.149637] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 396.330776] Lustre: lustre-OST0000: new disk, initializing [ 396.332806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 396.418391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.494995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 397.503832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 400.092225] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 409.327675] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 413.725431] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 413.995190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 417.684748] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 417.945578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 423.398042] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 423.401715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 423.406820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 423.442076] Lustre: Skipped 1 previous similar message [ 423.468356] Lustre: Skipped 4 previous similar messages [ 425.889737] Lustre: server umount lustre-OST0000 complete [ 433.123826] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 433.133402] LustreError: Skipped 1 previous similar message [ 433.150835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.567131] Lustre: server umount lustre-MDT0000 complete [ 438.756931] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.765560] LustreError: Skipped 1 previous similar message [ 439.472815] LustreError: 18002:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311730 with bad export cookie 2140644609837852394 [ 439.474225] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 439.493504] LustreError: 18002:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 443.816537] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:15:33 (1761311733) [ 450.287065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.680910] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.740715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.030092] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 461.282907] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 461.303214] LustreError: Skipped 1 previous similar message [ 462.531036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.877579] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 473.955399] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 478.971675] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 484.848971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 489.154437] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 495.882101] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 508.899790] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 508.908361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.918475] Lustre: Skipped 1 previous similar message [ 508.922739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 508.926849] Lustre: Skipped 2 previous similar messages [ 514.392513] Lustre: server umount lustre-OST0000 complete [ 514.398318] Lustre: Skipped 1 previous similar message [ 519.140461] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 526.309458] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.328716] LustreError: Skipped 1 previous similar message [ 526.969577] LustreError: 18707:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311818 with bad export cookie 2140644609837853248 [ 526.973068] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 526.984067] LustreError: 18707:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 536.878761] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 539.538793] Key type lgssc unregistered [ 539.936455] LNet: 22391:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 539.949759] LNet: Removed LNI 192.168.204.114@tcp [ 540.755446] Key type .llcrypt unregistered [ 540.758560] Key type ._llcrypt unregistered [ 560.270991] alg: No test for adler32 (adler32-zlib) [ 561.025447] Key type ._llcrypt registered [ 561.026684] Key type .llcrypt registered [ 561.120075] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 561.977329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 562.434993] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 562.610409] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 562.614782] LNet: Accept secure, port 988 [ 564.250376] Key type lgssc registered [ 565.338800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 573.494305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.928167] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.001210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.089577] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 580.081377] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.186374] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.701024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.233971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 591.378587] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 598.318990] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 603.521597] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 609.565287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 609.912246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 613.647103] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 620.914968] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 625.263938] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 634.859552] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 634.870864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.887291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 635.873637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 635.876378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.994177] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.001525] LustreError: Skipped 1 previous similar message [ 641.211523] Lustre: server umount lustre-OST0000 complete [ 646.116182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.131601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 646.148995] Lustre: Skipped 1 previous similar message [ 646.152652] Lustre: Skipped 1 previous similar message [ 651.239857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 651.241818] Lustre: Skipped 1 previous similar message [ 651.441156] Lustre: server umount lustre-MDT0000 complete [ 654.555755] LustreError: 23831:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311946 with bad export cookie 2369528743243806480 [ 654.564621] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.566758] LustreError: 23831:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 654.928428] Lustre: server umount lustre-MDT0001 complete [ 664.656053] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 667.445724] Key type lgssc unregistered [ 667.752541] LNet: 27853:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.769239] LNet: Removed LNI 192.168.204.114@tcp [ 668.585284] Key type .llcrypt unregistered [ 668.587424] Key type ._llcrypt unregistered [ 691.097139] alg: No test for adler32 (adler32-zlib) [ 691.853483] Key type ._llcrypt registered [ 691.859692] Key type .llcrypt registered [ 691.982660] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 693.088140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 693.739099] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 694.069233] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 694.073926] LNet: Accept secure, port 988 [ 695.775145] Key type lgssc registered [ 697.094759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 704.750699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 706.287521] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.335727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.032966] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 711.656029] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.769484] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 719.348945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 719.891838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 723.592136] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 730.762254] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 736.184557] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 749.359649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 749.605765] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.809853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 754.013227] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 757.230228] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 761.050777] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 766.949696] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 766.959107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.983098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 771.989341] Lustre: server umount lustre-OST0000 complete [ 776.163872] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 776.166704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.184195] Lustre: Skipped 1 previous similar message [ 776.186979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 776.195125] Lustre: Skipped 2 previous similar messages [ 777.696442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 777.697323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.254026] Lustre: server umount lustre-MDT0000 complete [ 782.817955] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 782.825768] LustreError: Skipped 1 previous similar message [ 784.758468] LustreError: 29315:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312076 with bad export cookie 1647962629211117902 [ 784.762329] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 784.768542] LustreError: 29315:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 785.067932] Lustre: server umount lustre-MDT0001 complete [ 794.993731] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 797.359183] Key type lgssc unregistered [ 797.629591] LNet: 33333:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.639438] LNet: Removed LNI 192.168.204.114@tcp [ 798.282466] Key type .llcrypt unregistered [ 798.284469] Key type ._llcrypt unregistered [ 815.419488] alg: No test for adler32 (adler32-zlib) [ 816.179454] Key type ._llcrypt registered [ 816.182866] Key type .llcrypt registered [ 816.250973] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 816.944288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 817.280944] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 817.440464] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 817.444680] LNet: Accept secure, port 988 [ 819.082487] Key type lgssc registered [ 819.924996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 825.617295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 826.906630] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 826.952445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.938319] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 831.968594] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 836.484173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 836.759952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 840.055513] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 845.862497] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 850.673435] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 856.786985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 857.130576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 860.655302] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 867.243667] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 876.597163] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 884.198368] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 884.210663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.220120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 886.760804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 886.768974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.184634] Lustre: server umount lustre-OST0000 complete [ 896.998450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.008688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.015663] Lustre: Skipped 2 previous similar messages [ 899.544799] Lustre: server umount lustre-MDT0000 complete [ 902.118766] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 902.140670] LustreError: Skipped 1 previous similar message [ 902.990173] LustreError: 36746:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312194 with bad export cookie 5260290021826770663 [ 902.994313] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 903.006192] LustreError: 36746:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 903.446260] Lustre: server umount lustre-MDT0001 complete [ 915.016519] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 918.139454] Key type lgssc unregistered [ 918.463572] LNet: 38475:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.475608] LNet: Removed LNI 192.168.204.114@tcp [ 919.568709] Key type .llcrypt unregistered [ 919.579684] Key type ._llcrypt unregistered [ 939.712057] alg: No test for adler32 (adler32-zlib) [ 940.472489] Key type ._llcrypt registered [ 940.473965] Key type .llcrypt registered [ 940.580692] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 941.680948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 942.063967] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 942.272287] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 942.275489] LNet: Accept secure, port 988 [ 943.935147] Key type lgssc registered [ 944.848764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 951.782365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.169093] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 953.263093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 957.045634] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 958.432959] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 963.555316] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.178410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 964.419643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 967.950486] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 975.553955] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 980.864762] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 985.882975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 986.085714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 989.987791] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 996.089424] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1003.505124] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1012.270739] Lustre: lustre-OST0000: Not available for connect from 192.168.204.14@tcp (stopping) [ 1013.728940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.737153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1013.737478] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1013.739846] Lustre: Skipped 1 previous similar message [ 1013.757046] Lustre: Skipped 1 previous similar message [ 1017.350557] Lustre: lustre-OST0000: Not available for connect from 192.168.204.14@tcp (stopping) [ 1017.731461] Lustre: server umount lustre-OST0000 complete [ 1035.234312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.244822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1035.250151] Lustre: Skipped 1 previous similar message [ 1040.352052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1041.131612] Lustre: server umount lustre-MDT0000 complete [ 1044.236577] LustreError: 39919:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312335 with bad export cookie 15366808403863051035 [ 1044.241907] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1050.768396] Lustre: server umount lustre-MDT0001 complete [ 1061.233625] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1063.961027] Key type lgssc unregistered [ 1064.195151] LNet: 43791:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.202611] LNet: Removed LNI 192.168.204.114@tcp [ 1064.987098] Key type .llcrypt unregistered [ 1064.988625] Key type ._llcrypt unregistered [ 1082.401203] alg: No test for adler32 (adler32-zlib) [ 1083.155357] Key type ._llcrypt registered [ 1083.159236] Key type .llcrypt registered [ 1083.262933] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1084.309151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1084.693829] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1084.860170] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1084.863197] LNet: Accept secure, port 988 [ 1086.519108] Key type lgssc registered [ 1087.492939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1094.217065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1095.490817] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.541794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.336379] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1100.771981] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.991518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1104.242412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1107.378360] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1113.783875] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1118.224153] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1123.138938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1123.481808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.550185] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 1126.876428] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1133.922629] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1140.198644] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1140.201818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.211766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1140.760760] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.14@tcp (stopping) [ 1141.596371] Lustre: server umount lustre-MDT0000 complete [ 1144.291389] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.294281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.648860] LustreError: 45233:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312436 with bad export cookie 10829243539647811683 [ 1144.652220] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1144.668959] LustreError: 45233:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1145.899734] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.14@tcp (stopping) [ 1149.409152] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.409979] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.422974] Lustre: Skipped 1 previous similar message [ 1149.431309] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1149.438728] LustreError: Skipped 3 previous similar messages [ 1151.095235] Lustre: server umount lustre-MDT0001 complete [ 1170.539387] Lustre: server umount lustre-OST0000 complete [ 1184.708498] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1186.901243] Key type lgssc unregistered [ 1187.114964] LNet: 49232:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1187.121959] LNet: Removed LNI 192.168.204.114@tcp [ 1187.871319] Key type .llcrypt unregistered [ 1187.872927] Key type ._llcrypt unregistered [ 1207.844070] alg: No test for adler32 (adler32-zlib) [ 1208.600983] Key type ._llcrypt registered [ 1208.602440] Key type .llcrypt registered [ 1208.746515] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1209.853782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1210.171356] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1210.357200] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1210.367190] LNet: Accept secure, port 988 [ 1212.039110] Key type lgssc registered [ 1212.847423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1218.660675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1235.231293] LustreError: 50670:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 1: rc = -5 [ 1266.975207] LustreError: 50670:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 1296.671166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1300.930381] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1307.159364] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1346.341683] Lustre: server umount lustre-OST0000 complete [ 1358.801661] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1360.951977] Key type lgssc unregistered [ 1361.180526] LNet: 52340:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.192450] LNet: Removed LNI 192.168.204.114@tcp [ 1361.795068] Key type .llcrypt unregistered [ 1361.803813] Key type ._llcrypt unregistered [ 1376.296130] alg: No test for adler32 (adler32-zlib) [ 1377.048425] Key type ._llcrypt registered [ 1377.050609] Key type .llcrypt registered [ 1377.113302] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1377.592044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1377.909353] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1378.038818] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1378.041619] LNet: Accept secure, port 988 [ 1379.655123] Key type lgssc registered [ 1380.443622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1385.348170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.618079] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1386.662715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1389.860897] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1392.114481] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.438997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.687182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1399.539766] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1405.041754] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1409.257312] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1413.191520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1413.412400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1416.354763] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1416.502245] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 1421.968309] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1426.726529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1426.732970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.791213] Lustre: server umount lustre-OST0000 complete [ 1437.156850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.162372] Lustre: Skipped 1 previous similar message [ 1437.163079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1437.177401] Lustre: Skipped 2 previous similar messages [ 1438.884959] Lustre: server umount lustre-MDT0000 complete [ 1441.449073] LustreError: 53779:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312732 with bad export cookie 1431204920188020213 [ 1441.454338] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1441.702746] Lustre: server umount lustre-MDT0001 complete [ 1450.192050] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1451.957311] Key type lgssc unregistered [ 1452.185685] LNet: 57450:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.190975] LNet: Removed LNI 192.168.204.114@tcp [ 1452.659984] Key type .llcrypt unregistered [ 1452.662600] Key type ._llcrypt unregistered [ 1467.596883] alg: No test for adler32 (adler32-zlib) [ 1468.349527] Key type ._llcrypt registered [ 1468.351239] Key type .llcrypt registered [ 1468.423075] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1468.976767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1469.343046] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1469.485451] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1469.488270] LNet: Accept secure, port 988 [ 1471.127158] Key type lgssc registered [ 1471.946779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1478.237990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1495.775516] LustreError: 58884:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 1: rc = -5 [ 1526.495187] LustreError: 58884:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 1556.255179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.856880] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1565.627466] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1570.112141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.008811] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.077580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.123748] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 1582.262077] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1584.096204] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1588.470659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1588.667468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1592.033354] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1598.116687] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1602.418460] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1604.064410] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1604.067663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.088392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1604.579027] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1604.586910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.598717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1606.113470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1606.121197] Lustre: Skipped 1 previous similar message [ 1611.233097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1611.240793] Lustre: Skipped 1 previous similar message [ 1616.355047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1616.358035] Lustre: Skipped 1 previous similar message [ 1618.399177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1618.549135] Lustre: server umount lustre-OST0000 complete [ 1638.057746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1638.099847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1638.110300] Lustre: Skipped 1 previous similar message [ 1644.339066] Lustre: server umount lustre-MDT0000 complete [ 1645.540458] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.546766] LustreError: Skipped 1 previous similar message [ 1647.325343] LustreError: 58890:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312938 with bad export cookie 4336774505639355751 [ 1647.329069] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1647.334438] LustreError: 58890:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1653.667161] Lustre: server umount lustre-MDT0001 complete [ 1662.990764] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1664.996316] Key type lgssc unregistered [ 1665.193931] LNet: 62754:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.203699] LNet: Removed LNI 192.168.204.114@tcp [ 1665.660182] Key type .llcrypt unregistered [ 1665.661754] Key type ._llcrypt unregistered [ 1681.460931] alg: No test for adler32 (adler32-zlib) [ 1682.219443] Key type ._llcrypt registered [ 1682.221487] Key type .llcrypt registered [ 1682.316983] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1683.066282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1683.368148] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1683.505054] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1683.509756] LNet: Accept secure, port 988 [ 1685.143156] Key type lgssc registered [ 1685.858875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1691.362966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.599472] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1692.635205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.217665] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1697.764995] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1700.366828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1700.547486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1703.256345] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1708.278727] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1712.504898] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1717.452867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1717.687983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1720.839642] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1722.801167] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 1726.154862] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1741.282450] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1741.289746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.298975] Lustre: Skipped 1 previous similar message [ 1741.302854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1741.305749] Lustre: Skipped 1 previous similar message [ 1743.375476] Lustre: server umount lustre-OST0000 complete [ 1746.912337] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1746.924085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.939739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1751.521467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.522415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1752.706354] Lustre: server umount lustre-MDT0000 complete [ 1756.078951] LustreError: 64194:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313047 with bad export cookie 6285003212066694807 [ 1756.078951] LustreError: 64195:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313047 with bad export cookie 6285003212066694807 [ 1756.080768] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1756.091041] LustreError: 64195:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1756.411815] Lustre: server umount lustre-MDT0001 complete [ 1766.008400] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1768.552751] Key type lgssc unregistered [ 1768.797444] LNet: 67889:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.812746] LNet: Removed LNI 192.168.204.114@tcp [ 1769.546662] Key type .llcrypt unregistered [ 1769.548218] Key type ._llcrypt unregistered [ 1795.130076] alg: No test for adler32 (adler32-zlib) [ 1795.882689] Key type ._llcrypt registered [ 1795.884996] Key type .llcrypt registered [ 1795.988831] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1797.043607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.296567] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1797.545601] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1797.551704] LNet: Accept secure, port 988 [ 1799.223128] Key type lgssc registered [ 1800.220757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1806.385841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.732375] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.776022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.332774] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1812.971297] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.066549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.082545] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.386413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1821.516654] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1827.177747] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1831.131119] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1836.012479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1836.269272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1838.316441] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:321 [ 1839.402328] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1845.171314] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1853.505827] Lustre: Failing over lustre-MDT0000 [ 1853.748744] Lustre: server umount lustre-MDT0000 complete [ 1854.431896] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1854.447992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.461619] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.021882] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1858.055301] Lustre: Skipped 1 previous similar message [ 1862.152712] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1862.169593] LustreError: Skipped 4 previous similar messages [ 1862.180895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.262188] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1862.262297] LustreError: 11-0: MGC192.168.204.114@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1862.273145] Lustre: Evicted from MGS (at 192.168.204.114@tcp) after server handle changed from 0xc6cf5b515ef0f389 to 0xc6cf5b515ef0f76b [ 1862.290782] LustreError: 72795:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.204.114@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000006fab3cef) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1862.306841] Lustre: MGC192.168.204.114@tcp: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 1862.394519] Lustre: *** cfs_fail_loc=135, val=0*** [ 1862.493011] BUG: spinlock bad magic on CPU#3, llog_process_th/72798 [ 1862.501748] lock: 0xffff9ab079cb98c0, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 1862.506965] CPU: 3 PID: 72798 Comm: llog_process_th Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1862.512317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1862.520969] Call Trace: [ 1862.523225] ? dump_stack+0xbb/0x10e [ 1862.524130] ? spin_dump.cold.3+0x32/0x65 [ 1862.525266] ? do_raw_spin_lock+0xce/0x190 [ 1862.526425] ? _raw_spin_lock+0x21/0x30 [ 1862.528163] ? mdd_changelog_fini+0x45/0x3a0 [mdd] [ 1862.529922] ? mdd_process_config+0x368/0x9c0 [mdd] [ 1862.531952] ? mdt_stack_fini+0x5de/0x1200 [mdt] [ 1862.534992] ? mdt_init0+0x650/0x1750 [mdt] [ 1862.536429] ? mdt_device_alloc+0x118/0x170 [mdt] [ 1862.538562] ? obd_setup+0x196/0x3e0 [obdclass] [ 1862.540360] ? class_setup+0x6ee/0x9e0 [obdclass] [ 1862.546503] ? class_process_config+0x17c7/0x2b90 [obdclass] [ 1862.549760] ? class_config_llog_handler+0xb73/0x1ab0 [obdclass] [ 1862.551817] ? llog_process_thread+0x109e/0x22a0 [obdclass] [ 1862.554393] ? keys_fill+0x10d/0x1b0 [obdclass] [ 1862.556453] ? lu_context_init+0xc8/0x240 [obdclass] [ 1862.561578] ? llog_process+0x40/0x40 [obdclass] [ 1862.563195] ? llog_process_thread_daemonize+0xe1/0x150 [obdclass] [ 1862.567132] ? kthread+0x1d1/0x200 [ 1862.568135] ? set_kthread_struct+0x70/0x70 [ 1862.569337] ? ret_from_fork+0x1f/0x30 [ 1862.572640] LustreError: 72798:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 1862.577210] LustreError: 72798:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.114@tcp: cfg command failed: rc = -2 [ 1862.582728] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1862.582728] [ 1862.590384] LustreError: 15c-8: MGC192.168.204.114@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1862.602615] LustreError: 72786:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1862.612610] LustreError: 72786:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 1862.632755] LustreError: 72786:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 1862.641206] LustreError: 69622:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313154 with bad export cookie 14325769344732165995 [ 1862.646146] LustreError: 72799:0:(ldlm_lock.c:2749:ldlm_lock_dump_handle()) ### ### ns: MGS lock: 000000004b0128f9/0xc6cf5b515ef0f78e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x40000000000000 nid: 0@lo remote: 0xc6cf5b515ef0f787 expref: 3 pid: 72793 timeout: 0 lvb_type: 0 [ 1862.659922] LustreError: 69622:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1862.699318] Lustre: server umount lustre-MDT0000 complete [ 1862.712361] LustreError: 72786:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1868.782228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.338846] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1870.358929] LustreError: Skipped 11 previous similar messages [ 1874.915863] Lustre: Evicted from MGS (at 192.168.204.114@tcp) after server handle changed from 0xc6cf5b515ef0f76b to 0xc6cf5b515ef0f805 [ 1874.929699] Lustre: MGC192.168.204.114@tcp: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 1875.161815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.203418] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1875.469570] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1878.870450] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1880.552832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1880.577052] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1880.616244] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:353 [ 1885.669927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.681488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1885.699665] Lustre: Skipped 1 previous similar message [ 1885.705443] Lustre: Skipped 1 previous similar message [ 1887.061497] Lustre: server umount lustre-OST0000 complete [ 1890.784898] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1890.787531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.789548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.798893] Lustre: Skipped 1 previous similar message [ 1895.911407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1895.913848] Lustre: Skipped 1 previous similar message [ 1896.555203] Lustre: server umount lustre-MDT0000 complete [ 1900.376961] LustreError: 69622:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313191 with bad export cookie 14325769344732166149 [ 1900.384244] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1900.388576] LustreError: 69622:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1900.403058] LustreError: Skipped 1 previous similar message [ 1900.852287] Lustre: server umount lustre-MDT0001 complete [ 1911.327496] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1913.806376] Key type lgssc unregistered [ 1914.105561] LNet: 74681:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.117163] LNet: Removed LNI 192.168.204.114@tcp [ 1914.828159] Key type .llcrypt unregistered [ 1914.829811] Key type ._llcrypt unregistered [ 1933.967407] alg: No test for adler32 (adler32-zlib) [ 1934.722067] Key type ._llcrypt registered [ 1934.724460] Key type .llcrypt registered [ 1934.833548] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1935.650317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1936.005655] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1936.179784] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1936.182378] LNet: Accept secure, port 988 [ 1937.825090] Key type lgssc registered [ 1938.495965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1943.751407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.073722] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1945.115067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.622669] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1950.179597] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.081560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.268322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1955.700773] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1961.131944] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1965.212180] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1970.288099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1970.490178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1973.786416] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 1974.576020] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:385 [ 1979.396269] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1987.572133] Lustre: Failing over lustre-MDT0000 [ 1987.783846] Lustre: server umount lustre-MDT0000 complete [ 1988.066250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.074312] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.082837] Lustre: Skipped 1 previous similar message [ 1991.216724] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1991.223268] LustreError: Skipped 2 previous similar messages [ 1995.167106] Lustre: 75457:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313279/real 1761313279] req@00000000886fe082 x1846870778326144/t0(0) o400->MGC192.168.204.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761313286 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1995.189354] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1995.819181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.293334] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.317982] LustreError: Skipped 6 previous similar messages [ 2001.383321] Lustre: Evicted from MGS (at 192.168.204.114@tcp) after server handle changed from 0xb9ad454248c3c2c9 to 0xb9ad454248c3c6b9 [ 2001.416547] Lustre: MGC192.168.204.114@tcp: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2001.574922] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2001.663271] LustreError: 79302:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-9) [ 2001.668407] LustreError: 79302:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.114@tcp: cfg command failed: rc = -9 [ 2001.672197] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2001.672197] [ 2001.682221] LustreError: 15c-8: MGC192.168.204.114@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2001.688162] LustreError: 79289:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2001.691162] LustreError: 79289:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -9 [ 2001.695393] LustreError: 79289:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 2001.718619] Lustre: server umount lustre-MDT0000 complete [ 2001.723614] LustreError: 79289:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2006.498584] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2006.510363] LustreError: Skipped 9 previous similar messages [ 2007.465742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.617978] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.620643] LustreError: 11-0: MGC192.168.204.114@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 2007.634112] LustreError: 76136:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2007.644140] Lustre: Evicted from MGS (at 192.168.204.114@tcp) after server handle changed from 0xb9ad454248c3c6b9 to 0xb9ad454248c3c75a [ 2007.653026] Lustre: MGC192.168.204.114@tcp: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2007.879672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.931821] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2009.137768] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2011.485121] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2013.158922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2013.185405] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2013.239794] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:417 [ 2018.785459] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2018.785907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.797690] LustreError: Skipped 1 previous similar message [ 2018.802629] Lustre: Skipped 1 previous similar message [ 2018.803587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2023.394681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2023.410940] Lustre: Skipped 1 previous similar message [ 2028.514564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2028.525129] Lustre: Skipped 1 previous similar message [ 2033.119287] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2033.266485] Lustre: server umount lustre-OST0000 complete [ 2038.754032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.755581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.756403] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2038.763397] Lustre: Skipped 2 previous similar messages [ 2038.776517] Lustre: Skipped 2 previous similar messages [ 2042.898639] Lustre: server umount lustre-MDT0000 complete [ 2043.873121] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.886103] LustreError: Skipped 5 previous similar messages [ 2045.854697] LustreError: 76124:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313337 with bad export cookie 13379426218978559834 [ 2045.862584] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2057.034919] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2059.289602] Key type lgssc unregistered [ 2059.514487] LNet: 81187:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.522632] LNet: Removed LNI 192.168.204.114@tcp [ 2060.054446] Key type .llcrypt unregistered [ 2060.056192] Key type ._llcrypt unregistered [ 2077.874851] alg: No test for adler32 (adler32-zlib) [ 2078.639221] Key type ._llcrypt registered [ 2078.640793] Key type .llcrypt registered [ 2078.769559] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2079.921201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2080.530379] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2080.870466] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2080.879257] LNet: Accept secure, port 988 [ 2082.575163] Key type lgssc registered [ 2083.999708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2092.015305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.580729] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.680821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.195768] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2099.179139] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2104.289186] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.069207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.408706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2110.298589] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2116.897751] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2121.512727] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2127.880144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2128.326153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.883621] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2139.089825] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2143.674659] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:449 [ 2152.931291] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2152.934199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.945043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2153.953832] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2153.954472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2153.967378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.981530] Lustre: Skipped 1 previous similar message [ 2155.501643] Lustre: server umount lustre-OST0000 complete [ 2159.073859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.076983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.080335] Lustre: Skipped 1 previous similar message [ 2159.085847] Lustre: Skipped 1 previous similar message [ 2164.195956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.202162] Lustre: Skipped 1 previous similar message [ 2164.851617] Lustre: server umount lustre-MDT0000 complete [ 2167.486170] LustreError: 82630:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313458 with bad export cookie 8926989155847930225 [ 2167.490560] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.504048] LustreError: 82630:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2167.819495] Lustre: server umount lustre-MDT0001 complete [ 2177.488600] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2179.798843] Key type lgssc unregistered [ 2180.059273] LNet: 86303:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.075652] LNet: Removed LNI 192.168.204.114@tcp [ 2180.908725] Key type .llcrypt unregistered [ 2180.913876] Key type ._llcrypt unregistered [ 2199.785570] alg: No test for adler32 (adler32-zlib) [ 2200.541374] Key type ._llcrypt registered [ 2200.542991] Key type .llcrypt registered [ 2200.633759] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2201.277988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.805748] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2201.967495] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2201.972208] LNet: Accept secure, port 988 [ 2203.639169] Key type lgssc registered [ 2204.690819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2210.900846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.408523] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.447433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.015919] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2217.442521] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.560674] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.921066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.297225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2226.556079] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2231.966736] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2236.500022] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2242.172841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2242.466791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.505696] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2251.808259] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2261.951257] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 2267.105686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2267.110770] Lustre: Skipped 1 previous similar message [ 2267.111620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2267.114239] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2267.128252] Lustre: Skipped 1 previous similar message [ 2269.211268] Lustre: server umount lustre-OST0000 complete [ 2274.786299] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2274.799919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.817145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2277.345201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2277.348593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.460228] Lustre: server umount lustre-MDT0000 complete [ 2281.130604] LustreError: 87760:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313572 with bad export cookie 891297256928809210 [ 2281.131574] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2281.150947] LustreError: 87760:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2281.457465] Lustre: server umount lustre-MDT0001 complete [ 2289.285978] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2291.102879] Key type lgssc unregistered [ 2291.339825] LNet: 91457:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2291.345619] LNet: Removed LNI 192.168.204.114@tcp [ 2292.126821] Key type .llcrypt unregistered [ 2292.128402] Key type ._llcrypt unregistered [ 2307.934878] alg: No test for adler32 (adler32-zlib) [ 2308.686397] Key type ._llcrypt registered [ 2308.690856] Key type .llcrypt registered [ 2308.775478] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2309.745417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.060451] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2310.245266] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2310.249823] LNet: Accept secure, port 988 [ 2311.930781] Key type lgssc registered [ 2313.127944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2319.601038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2320.972610] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.016873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2324.516314] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2326.499668] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.815353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.016059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2333.881694] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2338.434578] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2341.633152] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2345.291278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2345.462743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.857139] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2351.597864] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:513 [ 2352.386367] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2361.827987] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2361.830386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.845794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.850736] Lustre: Skipped 1 previous similar message [ 2364.938928] Lustre: server umount lustre-OST0000 complete [ 2367.178675] Lustre: server umount lustre-MDT0000 complete [ 2369.438992] LustreError: 92899:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313660 with bad export cookie 7022812698324789964 [ 2369.439136] LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 [ 2369.443965] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2369.446111] LustreError: Skipped 1 previous similar message [ 2375.807555] Lustre: server umount lustre-MDT0001 complete [ 2384.065224] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2386.208640] Key type lgssc unregistered [ 2386.476735] LNet: 96568:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2386.484952] LNet: Removed LNI 192.168.204.114@tcp [ 2387.131973] Key type .llcrypt unregistered [ 2387.134190] Key type ._llcrypt unregistered [ 2403.593291] alg: No test for adler32 (adler32-zlib) [ 2404.348550] Key type ._llcrypt registered [ 2404.350281] Key type .llcrypt registered [ 2404.416635] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2405.022366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2405.369616] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2405.547302] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2405.550458] LNet: Accept secure, port 988 [ 2407.207131] Key type lgssc registered [ 2408.200736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2414.672318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2431.199315] LustreError: 98004:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 1: rc = -5 [ 2462.943785] LustreError: 98004:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 2492.703292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.852595] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2500.665708] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2504.794055] Lustre: server umount lustre-OST0000 complete [ 2510.435311] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:50:00 (1761313800) [ 2514.845091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.072177] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.103492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.977047] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2520.545242] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2523.269041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.488382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2526.252533] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2531.282904] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2535.223840] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2539.551777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2539.763764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2542.188519] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2545.905779] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:545 [ 2547.160858] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2564.581859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2564.587407] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2564.587924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2564.592520] Lustre: Skipped 1 previous similar message [ 2569.433899] Lustre: server umount lustre-OST0000 complete [ 2574.817012] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2574.820918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.830538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2574.838066] Lustre: Skipped 1 previous similar message [ 2574.840880] Lustre: Skipped 1 previous similar message [ 2578.170688] Lustre: server umount lustre-MDT0000 complete [ 2579.939288] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.961109] LustreError: Skipped 1 previous similar message [ 2580.782985] LustreError: 99582:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313872 with bad export cookie 15319628420803447989 [ 2580.784681] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2580.789739] LustreError: 99582:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2581.091868] Lustre: server umount lustre-MDT0001 complete [ 2590.387670] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2592.470445] Key type lgssc unregistered [ 2592.654151] LNet: 103406:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2592.664835] LNet: Removed LNI 192.168.204.114@tcp [ 2593.301899] Key type .llcrypt unregistered [ 2593.303554] Key type ._llcrypt unregistered [ 2608.554066] alg: No test for adler32 (adler32-zlib) [ 2609.308414] Key type ._llcrypt registered [ 2609.310171] Key type .llcrypt registered [ 2609.376388] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2610.132890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2610.357834] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2610.497921] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2610.502814] LNet: Accept secure, port 988 [ 2612.135123] Key type lgssc registered [ 2612.973692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2618.157882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2619.388789] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.427875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2622.308798] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2624.484190] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.115410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.426850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2631.283339] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2636.688100] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2641.095273] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2646.716601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2646.985860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.092232] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:577 [ 2650.013677] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2656.321738] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2669.038838] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2669.048263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.070863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2673.005892] Lustre: server umount lustre-OST0000 complete [ 2679.776034] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2679.779256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.795743] Lustre: Skipped 2 previous similar messages [ 2679.796643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.800996] Lustre: Skipped 2 previous similar messages [ 2682.772195] Lustre: server umount lustre-MDT0000 complete [ 2684.902925] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.916020] LustreError: Skipped 1 previous similar message [ 2686.249245] LustreError: 104847:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313977 with bad export cookie 9815793891553379696 [ 2686.254722] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2686.589682] Lustre: server umount lustre-MDT0001 complete [ 2695.181916] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2697.081490] Key type lgssc unregistered [ 2697.284027] LNet: 108516:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2697.289763] LNet: Removed LNI 192.168.204.114@tcp [ 2697.980761] Key type .llcrypt unregistered [ 2697.982589] Key type ._llcrypt unregistered [ 2711.374490] alg: No test for adler32 (adler32-zlib) [ 2712.139996] Key type ._llcrypt registered [ 2712.145876] Key type .llcrypt registered [ 2712.275632] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2713.537516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2713.821977] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2714.109916] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2714.118958] LNet: Accept secure, port 988 [ 2715.799130] Key type lgssc registered [ 2716.679454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2722.430375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2739.999315] LustreError: 109808:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 1: rc = -5 [ 2771.743245] LustreError: 109808:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 2801.567207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.731813] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2810.088713] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2814.155361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.241060] LustreError: 15c-8: MGC192.168.204.114@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2824.247992] LustreError: 110794:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2824.253482] LustreError: 110794:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 2824.260245] LustreError: 110794:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 2824.358351] Lustre: server umount lustre-MDT0000 complete [ 2824.361371] LustreError: 110794:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2836.447332] Lustre: 109149:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314120/real 1761314120] req@00000000285a724d x1846871594108736/t0(0) o400->MGC192.168.204.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314127 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2836.465677] Lustre: 109149:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2836.650818] Lustre: server umount lustre-OST0000 complete [ 2844.222495] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 2850.476774] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2856.836263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2861.398045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2865.969114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2870.017971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2874.830136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2874.907297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.058556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2875.078875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2875.124239] Lustre: lustre-MDT0000: new disk, initializing [ 2875.178964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2875.190846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2878.530300] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2886.938564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2886.994850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.054400] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2887.080627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2887.083120] Lustre: Skipped 1 previous similar message [ 2887.130916] Lustre: lustre-MDT0001: new disk, initializing [ 2887.197123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2887.227871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2887.241522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2890.737056] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2898.341454] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2902.121936] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2906.956835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2907.015302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2907.019271] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2907.220762] Lustre: lustre-OST0000: new disk, initializing [ 2907.224593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2907.279672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.131087] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 2916.875913] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2917.435862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2917.446142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2919.883907] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2920.042929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2922.812454] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2922.959761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2927.586866] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2927.590150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2927.598330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2928.098513] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2928.098958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2928.103351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.421451] Lustre: server umount lustre-OST0000 complete [ 2933.217883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2933.219324] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2933.227280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2933.239393] Lustre: Skipped 2 previous similar messages [ 2938.340828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2938.343242] Lustre: Skipped 1 previous similar message [ 2939.163466] Lustre: server umount lustre-MDT0000 complete [ 2941.545487] LustreError: 113671:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314233 with bad export cookie 7731309301669675822 [ 2941.550524] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2941.556374] LustreError: 113671:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2947.589969] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:57:17 (1761314237) [ 2950.151472] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2961.204991] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 2967.741096] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2974.931231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2980.626312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2984.251609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2987.240933] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2991.223140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2991.266778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2991.397561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2991.425379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2991.478465] Lustre: lustre-MDT0000: new disk, initializing [ 2991.566200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2991.584064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2994.031870] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3000.804562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3000.868450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.923193] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3000.967244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3000.971136] Lustre: Skipped 1 previous similar message [ 3001.015831] Lustre: lustre-MDT0001: new disk, initializing [ 3001.108925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3001.163474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3001.170240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3004.275579] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3011.998784] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3016.221781] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3021.209907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3021.285233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3021.290244] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3021.611761] Lustre: lustre-OST0000: new disk, initializing [ 3021.613936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3021.706319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3025.200218] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3030.551190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3030.557514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3031.073411] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3033.801845] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3033.959284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3036.539459] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3036.655627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3040.737962] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3040.739441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3040.742090] LustreError: Skipped 1 previous similar message [ 3040.750667] Lustre: Skipped 1 previous similar message [ 3040.752251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3050.977433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3050.981473] Lustre: Skipped 4 previous similar messages [ 3051.999226] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3052.140316] Lustre: server umount lustre-OST0000 complete [ 3052.142472] Lustre: Skipped 1 previous similar message [ 3056.097871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3056.121038] Lustre: Skipped 2 previous similar messages [ 3060.765431] Lustre: server umount lustre-MDT0000 complete [ 3061.216185] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3061.227339] LustreError: Skipped 1 previous similar message [ 3063.365100] LustreError: 120801:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314354 with bad export cookie 7731309301669676676 [ 3063.365736] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3063.374532] LustreError: 120801:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3069.219809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.591161] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.653415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.457663] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3075.043352] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.863251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.036451] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3086.115300] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3089.339450] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3093.639953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.832134] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3102.447218] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3107.494496] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3110.372704] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3110.380071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.381329] LustreError: Skipped 1 previous similar message [ 3110.391745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3110.391753] Lustre: Skipped 2 previous similar messages [ 3115.827936] Lustre: server umount lustre-OST0000 complete [ 3115.830169] Lustre: Skipped 1 previous similar message [ 3127.108873] LustreError: 125945:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314418 with bad export cookie 7731309301669677530 [ 3127.114743] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.932532] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3136.586833] Key type lgssc unregistered [ 3136.762156] LNet: 128906:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3136.771378] LNet: Removed LNI 192.168.204.114@tcp [ 3137.256180] Key type .llcrypt unregistered [ 3137.258320] Key type ._llcrypt unregistered [ 3148.955384] alg: No test for adler32 (adler32-zlib) [ 3149.709425] Key type ._llcrypt registered [ 3149.715090] Key type .llcrypt registered [ 3149.789983] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 3160.061384] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3160.809821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3161.072138] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 3161.212306] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3161.216427] LNet: Accept secure, port 988 [ 3162.847115] Key type lgssc registered [ 3163.466708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3169.067304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3174.497515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3179.021323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3183.353231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3188.077904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3188.124485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.276838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3189.295911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3189.343219] Lustre: lustre-MDT0000: new disk, initializing [ 3189.411642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.432569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3192.067326] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3199.349832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3199.402172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.462217] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3199.494468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3199.497737] Lustre: Skipped 1 previous similar message [ 3199.542416] Lustre: lustre-MDT0001: new disk, initializing [ 3199.637271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3199.663462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3199.669972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3202.643301] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3208.879251] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3212.431650] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3216.866272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3216.910928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3216.915282] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3217.105201] Lustre: lustre-OST0000: new disk, initializing [ 3217.110025] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3217.165920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.176799] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3224.095047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3224.102861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3227.123796] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3230.505080] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3230.628419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3233.755274] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3233.912782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3239.394676] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3239.395732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.396702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3239.409029] Lustre: Skipped 1 previous similar message [ 3241.449125] Lustre: server umount lustre-OST0000 complete [ 3244.514540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.514540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.515387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3244.528720] Lustre: Skipped 2 previous similar messages [ 3249.634174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.756831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.765043] Lustre: Skipped 2 previous similar messages [ 3258.335711] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3258.409702] Lustre: server umount lustre-MDT0000 complete [ 3260.896153] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.074817] LustreError: 132189:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314553 with bad export cookie 3608728481453791598 [ 3262.079399] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3262.082838] LustreError: 132189:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3262.373781] Lustre: server umount lustre-MDT0001 complete [ 3270.378737] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:02:39 (1761314559) [ 3275.793230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3276.113103] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.122943] LustreError: Skipped 1 previous similar message [ 3276.165706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3279.499798] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3281.381690] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3286.498253] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.019333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3287.259885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3290.486924] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3295.424501] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3299.333427] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3301.538194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3301.542132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3301.551828] Lustre: Skipped 1 previous similar message [ 3301.554850] Lustre: Skipped 2 previous similar messages [ 3312.096231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3312.107247] Lustre: Skipped 4 previous similar messages [ 3315.172353] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3317.216800] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3317.238682] LustreError: Skipped 1 previous similar message [ 3317.300233] Lustre: server umount lustre-MDT0000 complete [ 3320.447557] LustreError: 137535:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314611 with bad export cookie 3608728481453792452 [ 3320.448623] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3320.458218] LustreError: 137535:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3326.865857] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:03:36 (1761314616) [ 3332.678187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3349.343554] LustreError: 139274:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 1: rc = -5 [ 3360.607571] LustreError: 139274:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 3389.407300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.845194] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3396.524374] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3397.683681] Lustre: server umount lustre-OST0000 complete [ 3397.689099] Lustre: Skipped 1 previous similar message [ 3403.042527] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:04:52 (1761314692) [ 3408.419604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3408.707949] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.759746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.378739] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3419.562276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.826300] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3429.463193] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3434.177806] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3439.898728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3443.676689] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3450.229850] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3466.208201] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3466.230412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3466.273756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3467.815186] Lustre: server umount lustre-MDT0000 complete [ 3468.775660] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3468.784300] LustreError: Skipped 4 previous similar messages [ 3470.715607] LustreError: 140658:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314762 with bad export cookie 3608728481453793117 [ 3470.727546] LustreError: 140658:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3470.728732] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3480.032975] Lustre: 143748:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314765/real 1761314765] req@00000000a0664413 x1846872062862400/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761314771 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3486.800956] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:06:16 (1761314776) [ 3491.444965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.800690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3491.804520] Lustre: Skipped 2 previous similar messages [ 3494.858461] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3501.490547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.264601] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3510.788318] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3515.200638] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3520.713305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3522.028677] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:65 [ 3524.502973] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3531.297460] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3535.579971] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3535.904480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3539.902583] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3540.097959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3542.497665] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3542.510527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.517708] LustreError: Skipped 1 previous similar message [ 3542.518986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3542.528479] Lustre: Skipped 3 previous similar messages [ 3556.320471] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3556.461810] Lustre: server umount lustre-OST0000 complete [ 3556.463980] Lustre: Skipped 2 previous similar messages [ 3562.978892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.985261] Lustre: Skipped 1 previous similar message [ 3573.215543] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3575.754304] LustreError: 144400:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314867 with bad export cookie 3608728481453794181 [ 3575.756141] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3575.759720] LustreError: 144400:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3582.461759] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:07:52 (1761314872) [ 3587.191916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3604.703610] LustreError: 148745:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 1: rc = -5 [ 3615.967476] LustreError: 148745:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 3644.769640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3644.772849] Lustre: Skipped 2 previous similar messages [ 3649.513872] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3656.389216] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3661.503040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3663.556275] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3663.569090] LustreError: Skipped 2 previous similar messages [ 3663.684322] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:97 [ 3666.729767] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3672.684879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3675.655310] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3681.236319] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3685.633472] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3689.969752] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3690.156942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3694.079743] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3694.290786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3698.656041] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3698.657827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3698.662657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3698.662666] Lustre: Skipped 11 previous similar messages [ 3698.701687] Lustre: Skipped 2 previous similar messages [ 3702.000683] Lustre: server umount lustre-MDT0000 complete [ 3702.002771] Lustre: Skipped 2 previous similar messages [ 3706.337884] LustreError: 148751:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314997 with bad export cookie 3608728481453795210 [ 3706.374292] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3717.087357] Lustre: 152458:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315002/real 1761315002] req@0000000026c9a3d7 x1846872062890688/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315008 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3725.187552] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:10:14 (1761315014) [ 3730.475519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3747.042719] LustreError: 153101:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 1: rc = -5 [ 3758.307765] LustreError: 153101:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 3787.103298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3787.109771] Lustre: Skipped 2 previous similar messages [ 3791.580929] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3798.194495] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3803.398344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.868183] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3806.893712] LustreError: Skipped 4 previous similar messages [ 3807.041378] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:129 [ 3810.431543] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3816.935711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.065639] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3825.372774] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3829.646248] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3834.163688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3834.214442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3834.218789] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3834.298235] Lustre: lustre-OST0001: new disk, initializing [ 3834.302158] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3837.153710] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3840.569671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3840.582365] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3845.122143] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3849.282943] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3849.510832] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3853.779426] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3853.991107] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3856.355867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3856.364309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3856.415362] Lustre: Skipped 1 previous similar message [ 3856.429694] Lustre: Skipped 3 previous similar messages [ 3862.268801] Lustre: server umount lustre-OST0000 complete [ 3862.270238] Lustre: Skipped 2 previous similar messages [ 3866.602221] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3884.920588] LustreError: 153109:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315176 with bad export cookie 3608728481453796155 [ 3884.927355] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3884.929796] LustreError: 153109:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3889.569305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3892.302786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3895.425729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3898.756841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3903.620823] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:13:13 (1761315193) [ 3904.789628] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3905.892845] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3907.113240] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:13:16 (1761315196) [ 3911.988834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.218458] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3912.245907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3915.703726] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3923.893494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3923.974672] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3924.011841] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3928.278395] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3934.960375] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3939.532978] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3944.529305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3944.641798] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3947.485884] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3952.569969] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3953.746182] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:161 [ 3956.480204] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3956.664948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3960.144732] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3960.335216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3963.874094] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3963.880661] LustreError: Skipped 1 previous similar message [ 3987.424532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3991.311687] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 3995.952366] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3995.972599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3995.987302] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 3996.001461] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:162 to 0x280000400:193 [ 3996.076548] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:193 [ 3997.643843] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4001.688679] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4001.877101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4005.308360] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4005.451607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4009.584782] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9ccdd83d0800.ost_server_uuid 40 [ 4010.777876] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ccdd83d0800.ost_server_uuid in FULL state after 0 sec [ 4010.931158] LustreError: 159684:0:(ldlm_lockd.c:721:ldlm_handle_ast_error()) ### client (nid 192.168.204.14@tcp) returned error from blocking AST (req@000000005a5434c6 x1846872062937152 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d4d4bc45/0x3214c76f1be37464 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.14@tcp remote: 0xf9e839723c0571e8 expref: 6 pid: 159683 timeout: 4110 lvb_type: 0 [ 4010.960906] LustreError: 138-a: lustre-MDT0000: A client on nid 192.168.204.14@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4010.975719] LustreError: 159672:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.14@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d4d4bc45/0x3214c76f1be37464 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.14@tcp remote: 0xf9e839723c0571e8 expref: 7 pid: 159683 timeout: 0 lvb_type: 0 [ 4016.608296] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4016.609034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.618314] LustreError: Skipped 1 previous similar message [ 4016.635563] Lustre: Skipped 7 previous similar messages [ 4019.971741] Lustre: server umount lustre-OST0000 complete [ 4019.973798] Lustre: Skipped 4 previous similar messages [ 4031.978584] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4031.988226] LustreError: Skipped 8 previous similar messages [ 4032.606598] LustreError: 159668:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315324 with bad export cookie 3608728481453797317 [ 4032.611210] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4032.621450] LustreError: 159668:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4049.480841] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4052.198715] Key type lgssc unregistered [ 4052.486720] LNet: 166065:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4052.498408] LNet: Removed LNI 192.168.204.114@tcp [ 4053.292779] Key type .llcrypt unregistered [ 4053.295336] Key type ._llcrypt unregistered [ 4071.901715] alg: No test for adler32 (adler32-zlib) [ 4072.654612] Key type ._llcrypt registered [ 4072.659705] Key type .llcrypt registered [ 4072.778704] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4073.576662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4073.807928] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 4074.023826] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 4074.030802] LNet: Accept secure, port 988 [ 4075.687145] Key type lgssc registered [ 4076.529661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4083.513823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4084.868244] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.904693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4088.966459] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4090.342420] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4095.460825] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4096.910334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4097.294170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4101.402137] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4108.933598] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4114.228612] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4120.640380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4121.073600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4125.199932] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4133.089217] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4135.481830] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 4143.302441] Lustre: Failing over lustre-MDT0000 [ 4143.537495] Lustre: server umount lustre-MDT0000 complete [ 4143.592573] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4143.606619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.644754] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4145.634198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4145.666896] Lustre: Skipped 1 previous similar message [ 4150.760806] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.765071] LustreError: Skipped 4 previous similar messages [ 4152.736125] Lustre: 166840:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315437/real 1761315437] req@000000001ff1329c x1846873020184128/t0(0) o400->MGC192.168.204.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315444 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4152.772213] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4158.960310] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.000346] LustreError: Skipped 10 previous similar messages [ 4163.342478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4170.214407] Lustre: Evicted from MGS (at 192.168.204.114@tcp) after server handle changed from 0xd20abca8c1c3d6a7 to 0xd20abca8c1c3da90 [ 4170.234498] Lustre: MGC192.168.204.114@tcp: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 4170.613859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4170.702930] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4175.130715] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4175.847109] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4175.849996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4177.379600] Lustre: lustre-MDT0000: Denying connection for new client 0fcea11b-4878-4856-9200-5004b96760b5 (at 192.168.204.14@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 4182.532774] Lustre: lustre-MDT0000: Denying connection for new client 0fcea11b-4878-4856-9200-5004b96760b5 (at 192.168.204.14@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 4188.642233] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4188.645846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.668310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4191.203832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4191.204225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.446635] Lustre: server umount lustre-OST0000 complete [ 4196.976848] LustreError: 171384:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4196.986813] Lustre: 170679:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4196.995612] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 4197.064087] Lustre: 170679:0:(mdt_handler.c:7532:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 [ 4201.441220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.450334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4201.454455] Lustre: Skipped 1 previous similar message [ 4203.438461] Lustre: server umount lustre-MDT0000 complete [ 4204.512196] Lustre: 166836:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315467/real 1761315467] req@0000000017e54378 x1846873020190016/t0(0) o400->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 2 to 1 dl 1761315496 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'' [ 4204.554994] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4204.561619] LustreError: Skipped 9 previous similar messages [ 4206.825618] LustreError: 167508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315498 with bad export cookie 15135116930723338896 [ 4206.832725] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4221.407171] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4221.713118] Lustre: server umount lustre-MDT0001 complete [ 4233.005988] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4235.915663] Key type lgssc unregistered [ 4236.166586] LNet: 172195:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4236.179982] LNet: Removed LNI 192.168.204.114@tcp [ 4237.031351] Key type .llcrypt unregistered [ 4237.035545] Key type ._llcrypt unregistered [ 4260.648196] alg: No test for adler32 (adler32-zlib) [ 4261.426572] Key type ._llcrypt registered [ 4261.428296] Key type .llcrypt registered [ 4261.503422] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4262.793262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4263.418973] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 4263.636758] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 4263.639611] LNet: Accept secure, port 988 [ 4265.303584] Key type lgssc registered [ 4266.521907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4274.262510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4275.721870] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.811873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.102313] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4280.815339] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4285.927487] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4288.081476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4288.325931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4292.253811] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4300.049324] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4305.919154] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4312.398673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4312.809993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.813914] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4322.423446] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4326.129842] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 4331.488516] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4331.494194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4331.500191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4331.500801] Lustre: Skipped 1 previous similar message [ 4332.801798] Lustre: server umount lustre-OST0000 complete [ 4336.612986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4336.619101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4336.622097] Lustre: Skipped 1 previous similar message [ 4336.624550] Lustre: Skipped 1 previous similar message [ 4341.734064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4341.736492] Lustre: Skipped 1 previous similar message [ 4342.378039] Lustre: server umount lustre-MDT0000 complete [ 4345.691606] LustreError: 173692:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315637 with bad export cookie 9243724603284782031 [ 4345.693258] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4345.723854] LustreError: 173692:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4346.083747] Lustre: server umount lustre-MDT0001 complete [ 4355.975686] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4358.397423] Key type lgssc unregistered [ 4358.728371] LNet: 177365:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4358.749846] LNet: Removed LNI 192.168.204.114@tcp [ 4359.537714] Key type .llcrypt unregistered [ 4359.550457] Key type ._llcrypt unregistered [ 4384.805492] alg: No test for adler32 (adler32-zlib) [ 4385.572810] Key type ._llcrypt registered [ 4385.578821] Key type .llcrypt registered [ 4385.720813] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4386.555989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4387.216359] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 4387.447857] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 4387.451246] LNet: Accept secure, port 988 [ 4389.103519] Key type lgssc registered [ 4390.126794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4397.605563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.208798] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4399.305434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4404.122852] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4404.706260] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4409.824941] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.671363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4413.132470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4417.589878] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4425.047692] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4430.447706] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4437.092979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4437.404356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4440.956477] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4447.841330] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4449.715568] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 4457.953463] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4457.956829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.964847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4458.984366] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4458.995796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4459.007678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4460.007065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4460.015563] Lustre: Skipped 1 previous similar message [ 4463.323501] Lustre: server umount lustre-OST0000 complete [ 4469.215883] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4469.222229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.236312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4472.982150] Lustre: server umount lustre-MDT0000 complete [ 4475.361297] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.361297] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.498442] LustreError: 179097:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315768 with bad export cookie 11253574663590625254 [ 4476.500600] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.520181] LustreError: 179097:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4476.854820] Lustre: server umount lustre-MDT0001 complete [ 4487.978979] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4490.466475] Key type lgssc unregistered [ 4490.719169] LNet: 182774:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4490.732980] LNet: Removed LNI 192.168.204.114@tcp [ 4491.528244] Key type .llcrypt unregistered [ 4491.531140] Key type ._llcrypt unregistered [ 4509.332661] alg: No test for adler32 (adler32-zlib) [ 4510.087461] Key type ._llcrypt registered [ 4510.089338] Key type .llcrypt registered [ 4510.218213] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4511.175796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4511.448351] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 4511.653596] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 4511.655640] LNet: Accept secure, port 988 [ 4513.313927] Key type lgssc registered [ 4514.437188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4523.464394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.776943] Lustre: *** cfs_fail_loc=135, val=0*** [ 4524.859856] LustreError: 184348:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 4524.862719] LustreError: 184348:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.114@tcp: cfg command failed: rc = -2 [ 4524.879538] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4524.879538] [ 4524.920542] LustreError: 15c-8: MGC192.168.204.114@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4524.925604] LustreError: 184309:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4524.928728] LustreError: 184309:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 4524.931974] LustreError: 184309:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 4531.167229] Lustre: 184309:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315816/real 1761315816] req@000000005f813cd1 x1846873479447360/t0(0) o251->MGC192.168.204.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315822 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4531.301369] Lustre: server umount lustre-MDT0000 complete [ 4531.303814] LustreError: 184309:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4536.804782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4537.136027] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4537.201168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4541.375235] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4542.437430] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4547.559379] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4549.644398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4550.261650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4554.784185] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4561.939513] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4566.039368] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4570.601109] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4570.610932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.638304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4572.130583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4572.130899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4572.141614] Lustre: Skipped 1 previous similar message [ 4577.248643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4577.253181] Lustre: Skipped 1 previous similar message [ 4580.323099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4580.335086] Lustre: Skipped 1 previous similar message [ 4582.367159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4582.539527] Lustre: server umount lustre-MDT0000 complete [ 4585.440927] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.335763] LustreError: 184702:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315877 with bad export cookie 10922140511668869482 [ 4586.367543] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4586.839692] Lustre: server umount lustre-MDT0001 complete [ 4597.167918] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4600.586564] Key type lgssc unregistered [ 4600.987097] LNet: 187206:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4601.003972] LNet: Removed LNI 192.168.204.114@tcp [ 4601.917546] Key type .llcrypt unregistered [ 4601.920748] Key type ._llcrypt unregistered [ 4626.406959] alg: No test for adler32 (adler32-zlib) [ 4627.162508] Key type ._llcrypt registered [ 4627.165044] Key type .llcrypt registered [ 4627.305348] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4627.538710] Key type .llcrypt unregistered [ 4627.539900] Key type ._llcrypt unregistered [ 4641.043715] alg: No test for adler32 (adler32-zlib) [ 4641.798405] Key type ._llcrypt registered [ 4641.806822] Key type .llcrypt registered [ 4641.938539] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4642.594848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4643.187329] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 4643.397658] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 4643.406804] LNet: Accept secure, port 988 [ 4645.079110] Key type lgssc registered [ 4646.154822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4655.010892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4672.735374] LustreError: 189462:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 1: rc = -5 [ 4703.455191] LustreError: 189462:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 4733.215624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4737.594518] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4743.937837] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4750.056279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.047808] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4755.134293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4755.202824] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:321 [ 4758.823759] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4760.551845] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4765.664652] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4766.106172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4766.409802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4770.643044] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4776.716676] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4781.202895] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4783.674679] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 4796.385436] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4796.391985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4796.400790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4797.412465] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4797.419064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4797.427776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4798.436607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4798.455583] Lustre: Skipped 1 previous similar message [ 4800.280988] Lustre: server umount lustre-OST0000 complete [ 4803.552145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4803.558743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4803.564336] Lustre: Skipped 1 previous similar message [ 4803.580259] Lustre: Skipped 1 previous similar message [ 4808.672581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4808.678421] Lustre: Skipped 1 previous similar message [ 4809.691712] Lustre: server umount lustre-MDT0000 complete [ 4812.935058] LustreError: 189468:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316104 with bad export cookie 4494019405664339541 [ 4812.939610] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4812.943591] LustreError: 189468:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4813.293789] Lustre: server umount lustre-MDT0001 complete [ 4821.844288] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4824.441301] Key type lgssc unregistered [ 4824.749879] LNet: 193599:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4824.754940] LNet: Removed LNI 192.168.204.114@tcp [ 4825.453871] Key type .llcrypt unregistered [ 4825.457847] Key type ._llcrypt unregistered [ 4843.941366] alg: No test for adler32 (adler32-zlib) [ 4844.694368] Key type ._llcrypt registered [ 4844.700117] Key type .llcrypt registered [ 4844.777629] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4845.791511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4846.077304] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 4846.232986] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 4846.236712] LNet: Accept secure, port 988 [ 4847.879203] Key type lgssc registered [ 4848.745158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4855.382277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4856.771913] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4856.841269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4861.373891] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4861.924947] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4867.042922] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4869.682076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4870.026644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4873.733892] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4880.998936] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4886.343689] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4892.202701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4892.431355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4895.444872] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4900.627254] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4906.802774] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:353 [ 4914.747398] Lustre: Failing over lustre-MDT0000 [ 4914.979485] Lustre: server umount lustre-MDT0000 complete [ 4916.192089] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4916.207299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.235276] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4917.219230] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4917.242409] Lustre: Skipped 1 previous similar message [ 4921.860467] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.878045] LustreError: Skipped 4 previous similar messages [ 4924.383167] Lustre: 194373:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316208/real 1761316208] req@000000004702517e x1846873829684736/t0(0) o400->MGC192.168.204.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761316215 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4924.428829] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4930.052498] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4930.063749] LustreError: Skipped 11 previous similar messages [ 4933.398598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4940.771944] Lustre: Evicted from MGS (at 192.168.204.114@tcp) after server handle changed from 0xbc1491ab140797a9 to 0xbc1491ab14079b84 [ 4940.781702] Lustre: MGC192.168.204.114@tcp: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 4940.927138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4940.967114] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4944.359675] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 4945.413832] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4946.406734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4946.434990] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4946.480344] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:385 [ 4948.981631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire in log lustre-MDT0000 [ 4956.515843] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight in log lustre-client [ 4966.368511] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4966.372704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4966.382772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4966.886042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4969.442196] Lustre: server umount lustre-OST0000 complete [ 4977.120159] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4977.121577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4977.131630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4977.143937] Lustre: Skipped 2 previous similar messages [ 4978.625061] Lustre: server umount lustre-MDT0000 complete [ 4981.989222] LustreError: 195042:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316273 with bad export cookie 13552617342626143108 [ 4981.990157] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4981.996869] LustreError: 195042:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4982.434973] Lustre: server umount lustre-MDT0001 complete [ 4991.965370] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4994.657277] Key type lgssc unregistered [ 4994.915821] LNet: 200191:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4994.925366] LNet: Removed LNI 192.168.204.114@tcp [ 4995.591309] Key type .llcrypt unregistered [ 4995.601051] Key type ._llcrypt unregistered [ 5014.040908] alg: No test for adler32 (adler32-zlib) [ 5014.795208] Key type ._llcrypt registered [ 5014.808810] Key type .llcrypt registered [ 5014.888930] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5015.491652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5015.770773] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 5015.919475] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 5015.921946] LNet: Accept secure, port 988 [ 5017.559492] Key type lgssc registered [ 5018.405936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5024.905654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.234525] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.286628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5029.715644] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5031.407800] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5036.519948] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5037.443802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5037.782219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5041.849513] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5048.348129] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5053.434642] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5058.938551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5059.202921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5062.470092] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5068.373413] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5075.567797] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:417 [ 5080.386372] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5087.988065] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5098.407702] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5111.780693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5111.787425] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5111.787477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5111.795187] Lustre: Skipped 1 previous similar message [ 5116.820398] Lustre: server umount lustre-OST0000 complete [ 5119.969391] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5119.977434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5119.985962] Lustre: Skipped 1 previous similar message [ 5119.990423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5122.025485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5122.029256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5122.046308] Lustre: Skipped 1 previous similar message [ 5125.796819] Lustre: server umount lustre-MDT0000 complete [ 5127.138271] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5127.150883] LustreError: Skipped 1 previous similar message [ 5128.904330] LustreError: 204274:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316420 with bad export cookie 17861476883957132371 [ 5128.910705] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5128.914291] LustreError: 204274:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5129.210931] Lustre: server umount lustre-MDT0001 complete [ 5139.518586] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 5141.676531] Key type lgssc unregistered [ 5141.908356] LNet: 205529:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5141.923399] LNet: Removed LNI 192.168.204.114@tcp [ 5142.612514] Key type .llcrypt unregistered [ 5142.614595] Key type ._llcrypt unregistered [ 5158.768760] alg: No test for adler32 (adler32-zlib) [ 5159.528521] Key type ._llcrypt registered [ 5159.533803] Key type .llcrypt registered [ 5159.627860] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5160.499033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5160.916113] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 5161.096070] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 5161.101884] LNet: Accept secure, port 988 [ 5162.751131] Key type lgssc registered [ 5163.708680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5170.750074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5172.078112] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.129588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5175.752853] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5177.318024] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5182.436050] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5183.187105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.470758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5186.456291] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5192.518711] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5198.664601] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5205.034450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5205.322483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5209.355425] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5216.424297] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5221.680186] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:449 [ 5225.799399] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 5237.843974] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 5248.810819] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 5257.632539] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 5273.060797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.067795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5273.071574] Lustre: Skipped 1 previous similar message [ 5273.083983] Lustre: Skipped 1 previous similar message [ 5277.471316] Lustre: server umount lustre-OST0000 complete [ 5283.297618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.299403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5283.308667] Lustre: Skipped 1 previous similar message [ 5283.315157] Lustre: Skipped 1 previous similar message [ 5287.356517] Lustre: server umount lustre-MDT0000 complete [ 5288.420259] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.438752] LustreError: Skipped 1 previous similar message [ 5290.778288] LustreError: 210184:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316582 with bad export cookie 4530210536292255379 [ 5290.784792] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5290.796434] LustreError: 210184:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5291.215372] Lustre: server umount lustre-MDT0001 complete [ 5301.695383] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 5303.889990] Key type lgssc unregistered [ 5304.159111] LNet: 212378:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5304.166508] LNet: Removed LNI 192.168.204.114@tcp [ 5304.829202] Key type .llcrypt unregistered [ 5304.830543] Key type ._llcrypt unregistered [ 5321.103204] alg: No test for adler32 (adler32-zlib) [ 5321.855444] Key type ._llcrypt registered [ 5321.864026] Key type .llcrypt registered [ 5321.958477] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5322.924367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5323.253530] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 5323.516559] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 5323.522414] LNet: Accept secure, port 988 [ 5325.223243] Key type lgssc registered [ 5326.382842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5333.699610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5335.115910] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5335.180508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.652223] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5340.641420] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5345.659855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5345.768825] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5345.953311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5349.488083] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5356.045530] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5361.039625] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5366.700336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5366.989831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5370.654375] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5377.073461] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5386.509962] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:481 [ 5397.255337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5397.574801] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5397.930789] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5403.982827] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5410.680667] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5412.195838] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5423.749298] Lustre: Permanently deactivating lustre-OST0001 [ 5423.762862] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 5433.384768] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5447.640646] Lustre: Permanently reactivating lustre-OST0001 [ 5447.662395] Lustre: Modifying parameter lustre-OST0001-osc.osc.active in log lustre-client [ 5447.667366] Lustre: Skipped 2 previous similar messages [ 5456.447712] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5456.454465] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5456.459673] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5456.472647] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 5456.475888] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:34 to 0x2c0000400:65 [ 5456.510609] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5461.984312] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5461.984512] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5461.987977] LustreError: Skipped 2 previous similar messages [ 5461.988647] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5461.999806] Lustre: Skipped 2 previous similar messages [ 5462.003767] LustreError: 217188:0:(osp_precreate.c:677:osp_precreate_send()) lustre-OST0001-osc-MDT0000: can't precreate: rc = -107 [ 5463.012918] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5463.016780] Lustre: Skipped 2 previous similar messages [ 5464.398462] Lustre: server umount lustre-OST0001 complete [ 5468.137486] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.639947] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5468.648853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.660193] LustreError: Skipped 1 previous similar message [ 5468.669881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5468.696600] Lustre: Skipped 1 previous similar message [ 5473.256794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5473.257427] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5473.269775] Lustre: Skipped 2 previous similar messages [ 5473.280054] LustreError: Skipped 1 previous similar message [ 5474.664643] Lustre: server umount lustre-OST0000 complete [ 5478.368446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5483.493973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5483.496815] Lustre: Skipped 3 previous similar messages [ 5484.490957] Lustre: server umount lustre-MDT0000 complete [ 5488.198563] LustreError: 215796:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316779 with bad export cookie 14040424643297003054 [ 5488.205547] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5488.208335] LustreError: 215796:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5488.605576] Lustre: server umount lustre-MDT0001 complete [ 5500.201257] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 5502.970832] Key type lgssc unregistered [ 5503.344361] LNet: 219012:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5503.360076] LNet: Removed LNI 192.168.204.114@tcp [ 5504.297376] Key type .llcrypt unregistered [ 5504.299305] Key type ._llcrypt unregistered [ 5533.323425] alg: No test for adler32 (adler32-zlib) [ 5534.080371] Key type ._llcrypt registered [ 5534.086938] Key type .llcrypt registered [ 5534.244526] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 5545.327884] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5546.375543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5546.672717] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 5546.862316] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 5546.866824] LNet: Accept secure, port 988 [ 5548.520412] Key type lgssc registered [ 5549.435416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5556.864213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5562.612554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5567.808952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5573.500115] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5580.088832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5580.188572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.557963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5581.588704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5581.664809] Lustre: lustre-MDT0000: new disk, initializing [ 5581.775698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5581.799607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5585.397904] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5595.795487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5595.879145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5595.995620] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5596.039252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5596.042320] Lustre: Skipped 1 previous similar message [ 5596.090656] Lustre: lustre-MDT0001: new disk, initializing [ 5596.179793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5596.230700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5596.239982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5599.927302] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5608.038893] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5612.415253] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5618.214592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5618.316556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5618.332819] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5618.638815] Lustre: lustre-OST0000: new disk, initializing [ 5618.643507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5618.743095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5622.587877] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5624.363116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5624.373072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5631.131615] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5635.442492] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5635.680286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5639.658795] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5639.907826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5644.769641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5644.776313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5644.778914] Lustre: Skipped 1 previous similar message [ 5647.828986] Lustre: server umount lustre-OST0000 complete [ 5652.449113] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5652.452240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.479785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5652.482031] Lustre: Skipped 1 previous similar message [ 5655.009686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5655.011864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.429082] Lustre: server umount lustre-MDT0000 complete [ 5660.133832] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5660.151882] LustreError: Skipped 1 previous similar message [ 5661.001182] LustreError: 226787:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316952 with bad export cookie 1772714350043495876 [ 5661.003986] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5661.019199] LustreError: 226787:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5661.494952] Lustre: server umount lustre-MDT0001 complete [ 5669.783730] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 10:42:39 (1761316959) [ 5675.451509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5676.002865] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5676.094392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5680.640963] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5681.122536] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.130487] LustreError: Skipped 1 previous similar message [ 5686.242065] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.742606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5689.068018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5692.749872] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5698.978773] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5704.281957] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5710.526058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5710.873526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5715.269104] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5721.832134] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5728.731772] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5735.966381] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5742.279422] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5754.243898] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5754.250580] Lustre: Skipped 1 previous similar message [ 5774.331890] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5774.335990] Lustre: Skipped 2 previous similar messages [ 5809.313469] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5809.317655] Lustre: Skipped 3 previous similar messages [ 5879.770048] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5879.773810] Lustre: Skipped 8 previous similar messages [ 5895.138584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.142656] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5895.149786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5895.149794] Lustre: Skipped 1 previous similar message [ 5899.963744] Lustre: server umount lustre-OST0000 complete [ 5904.352513] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5904.366360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.373569] Lustre: Skipped 1 previous similar message [ 5904.382285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5904.385069] Lustre: Skipped 1 previous similar message [ 5908.884166] Lustre: server umount lustre-MDT0000 complete [ 5910.505267] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5911.956638] LustreError: 230530:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317203 with bad export cookie 1772714350043496730 [ 5911.962158] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5911.971736] LustreError: 230530:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5922.265717] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 5924.575893] Key type lgssc unregistered [ 5924.920109] LNet: 232531:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5924.930137] LNet: Removed LNI 192.168.204.114@tcp [ 5925.600846] Key type .llcrypt unregistered [ 5925.602525] Key type ._llcrypt unregistered [ 5945.812570] alg: No test for adler32 (adler32-zlib) [ 5946.571981] Key type ._llcrypt registered [ 5946.573496] Key type .llcrypt registered [ 5946.759804] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5947.570341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5947.923993] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 5948.116588] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 5948.121240] LNet: Accept secure, port 988 [ 5949.792067] Key type lgssc registered [ 5950.809787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5957.631407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5958.920022] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.994453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5961.944200] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5964.257136] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.681905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.990331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5972.285840] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5978.450233] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5982.726084] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5987.192636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5987.480237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5990.838500] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all 8 [ 5995.864885] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6010.335888] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6010.345553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.349674] LustreError: Skipped 1 previous similar message [ 6010.357306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6010.378709] Lustre: Skipped 1 previous similar message [ 6014.489622] Lustre: server umount lustre-OST0000 complete [ 6020.067088] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6020.074149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6020.092966] Lustre: Skipped 1 previous similar message [ 6020.098827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6023.635503] Lustre: server umount lustre-MDT0000 complete [ 6025.698352] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6025.730236] LustreError: Skipped 1 previous similar message [ 6027.095174] LustreError: 233973:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317318 with bad export cookie 2682011931896292560 [ 6027.101701] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6027.532926] Lustre: server umount lustre-MDT0001 complete [ 6038.877493] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 6042.280774] Key type lgssc unregistered [ 6042.550373] LNet: 237808:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6043.617187] LNet: Removed LNI 192.168.204.114@tcp [ 6044.574806] Key type .llcrypt unregistered [ 6044.581213] Key type ._llcrypt unregistered [ 6067.185476] alg: No test for adler32 (adler32-zlib) [ 6067.939395] Key type ._llcrypt registered [ 6067.940931] Key type .llcrypt registered [ 6068.010338] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 6068.207938] Key type .llcrypt unregistered [ 6068.210171] Key type ._llcrypt unregistered [ 6090.819556] alg: No test for adler32 (adler32-zlib) [ 6091.586475] Key type ._llcrypt registered [ 6091.588179] Key type .llcrypt registered [ 6091.691080] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 6106.642542] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6107.929841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6108.592672] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 6108.832557] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 6108.853962] LNet: Accept secure, port 988 [ 6110.531782] Key type lgssc registered [ 6111.561789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6122.014424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6127.600971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6132.861041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6138.819444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6140.298847] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5889 sec ========== 10:50:29 (1761317429) [ 6171.425724] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 6174.427289] Key type lgssc unregistered [ 6174.676462] LNet: 244060:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6174.691259] LNet: Removed LNI 192.168.204.114@tcp [ 6175.460103] Key type .llcrypt unregistered [ 6175.469425] Key type ._llcrypt unregistered