[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468779745 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.002420] x2apic enabled [ 0.003014] Switched APIC routing to physical x2apic. [ 0.004025] kvm-guest: setup PV IPIs [ 0.007592] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.008032] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.011120] LSM: Security Framework initializing [ 0.012075] Yama: becoming mindful. [ 0.013063] SELinux: Initializing. [ 0.014109] *** VALIDATE selinux *** [ 0.023344] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029026] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030204] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031154] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032178] *** VALIDATE tmpfs *** [ 0.033592] *** VALIDATE proc *** [ 0.034351] *** VALIDATE cgroup *** [ 0.035016] *** VALIDATE cgroup2 *** [ 0.037586] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041034] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.046356] debug: unmapping init [mem 0xffffffffb0259000-0xffffffffb0260fff] [ 0.049248] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050750] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051042] ... version: 2 [ 0.052020] ... bit width: 48 [ 0.053017] ... generic registers: 4 [ 0.054013] ... value mask: 0000ffffffffffff [ 0.055021] ... max period: 00007fffffffffff [ 0.056025] ... fixed-purpose events: 3 [ 0.057023] ... event mask: 000000070000000f [ 0.059308] rcu: Hierarchical SRCU implementation. [ 0.061862] smp: Bringing up secondary CPUs ... [ 0.062761] x86: Booting SMP configuration: [ 0.063043] .... node #0, CPUs: #1 #2 #3 [ 0.071521] smp: Brought up 1 node, 4 CPUs [ 0.073025] smpboot: Max logical packages: 1 [ 0.074028] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.148434] node 0 deferred pages initialised in 69ms [ 0.157149] devtmpfs: initialized [ 0.158000] x86/mm: Memory block size: 128MB [ 0.161161] gcov: version magic: 0x41383552 [ 0.169417] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.180231] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.189242] pinctrl core: initialized pinctrl subsystem [ 0.193260] [ 0.194013] ************************************************************* [ 0.197016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.201016] ** ** [ 0.206022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.211022] ** ** [ 0.217024] ** This means that this kernel is built to expose internal ** [ 0.221022] ** IOMMU data structures, which may compromise security on ** [ 0.225023] ** your system. ** [ 0.228028] ** ** [ 0.235024] ** If you see this message and you are not debugging the ** [ 0.237018] ** kernel, report this immediately to your vendor! ** [ 0.241022] ** ** [ 0.244045] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.248030] ************************************************************* [ 0.265270] NET: Registered protocol family 16 [ 0.270659] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.275205] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.284177] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.290177] cpuidle: using governor menu [ 0.294288] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.299215] PCI: Using configuration type 1 for base access [ 0.306189] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.332098] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.335042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.350486] cryptd: max_cpu_qlen set to 1000 [ 0.360175] ACPI: Added _OSI(Module Device) [ 0.369039] ACPI: Added _OSI(Processor Device) [ 0.372023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.380023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.398311] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.421191] ACPI: Interpreter enabled [ 0.425103] ACPI: PM: (supports S0 S3 S4 S5) [ 0.427020] ACPI: Using IOAPIC for interrupt routing [ 0.432149] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.437753] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.448123] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.450062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.452028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.460108] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.467841] acpiphp: Slot [2] registered [ 0.469161] acpiphp: Slot [5] registered [ 0.471163] acpiphp: Slot [6] registered [ 0.473162] acpiphp: Slot [7] registered [ 0.475140] acpiphp: Slot [8] registered [ 0.477147] acpiphp: Slot [9] registered [ 0.479149] acpiphp: Slot [10] registered [ 0.480167] acpiphp: Slot [3] registered [ 0.482169] acpiphp: Slot [4] registered [ 0.484148] acpiphp: Slot [11] registered [ 0.486124] acpiphp: Slot [12] registered [ 0.488119] acpiphp: Slot [13] registered [ 0.489149] acpiphp: Slot [14] registered [ 0.490124] acpiphp: Slot [15] registered [ 0.491190] acpiphp: Slot [16] registered [ 0.493138] acpiphp: Slot [17] registered [ 0.495135] acpiphp: Slot [18] registered [ 0.497138] acpiphp: Slot [19] registered [ 0.499148] acpiphp: Slot [20] registered [ 0.501173] acpiphp: Slot [21] registered [ 0.503130] acpiphp: Slot [22] registered [ 0.504252] acpiphp: Slot [23] registered [ 0.506151] acpiphp: Slot [24] registered [ 0.508118] acpiphp: Slot [25] registered [ 0.510124] acpiphp: Slot [26] registered [ 0.512198] acpiphp: Slot [27] registered [ 0.514116] acpiphp: Slot [28] registered [ 0.516311] acpiphp: Slot [29] registered [ 0.518350] acpiphp: Slot [30] registered [ 0.520145] acpiphp: Slot [31] registered [ 0.522098] PCI host bridge to bus 0000:00 [ 0.524037] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.527027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.528028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.531154] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.534059] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.537028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.540228] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.543286] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.547895] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.565923] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.570067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.573024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.577025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.580030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.582643] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.585108] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.588065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.592435] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.608027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.633022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.649022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.664000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.684025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.712025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.737230] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.749035] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.760022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.770022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.796024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.808000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.813020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.821022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.849025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.865105] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.889030] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.916024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.940024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.952813] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.969023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.983201] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.009024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.030000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.040020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.051021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.076025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.097297] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.109382] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.115484] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.119131] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.121572] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.132139] iommu: Default domain type: Passthrough [ 1.135604] SCSI subsystem initialized [ 1.137215] ACPI: bus type USB registered [ 1.138000] usbcore: registered new interface driver usbfs [ 1.144247] usbcore: registered new interface driver hub [ 1.145349] usbcore: registered new device driver usb [ 1.148186] pps_core: LinuxPPS API ver. 1 registered [ 1.152018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.163098] PTP clock support registered [ 1.166122] EDAC MC: Ver: 3.0.0 [ 1.169338] PCI: Using ACPI for IRQ routing [ 1.176711] NetLabel: Initializing [ 1.183025] NetLabel: domain hash size = 128 [ 1.191018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.195138] NetLabel: unlabeled traffic allowed by default [ 1.210074] vgaarb: loaded [ 1.213141] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.215023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.229530] clocksource: Switched to clocksource kvm-clock [ 1.433684] VFS: Disk quotas dquot_6.6.0 [ 1.443434] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.451227] *** VALIDATE ramfs *** [ 1.454667] *** VALIDATE hugetlbfs *** [ 1.459791] pnp: PnP ACPI init [ 1.467947] pnp: PnP ACPI: found 6 devices [ 1.545525] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.550601] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.557777] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.568299] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.580627] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.589921] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.600236] NET: Registered protocol family 2 [ 1.604907] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.620818] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.634763] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.660851] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.668642] TCP: Hash tables configured (established 65536 bind 65536) [ 1.677748] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.691296] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.698277] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.707437] NET: Registered protocol family 1 [ 1.713231] RPC: Registered named UNIX socket transport module. [ 1.720247] RPC: Registered udp transport module. [ 1.728033] RPC: Registered tcp transport module. [ 1.734259] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.739654] NET: Registered protocol family 44 [ 1.741381] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.744213] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.749198] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.755674] PCI: CLS 0 bytes, default 64 [ 1.760127] Unpacking initramfs... [ 4.059616] debug: unmapping init [mem 0xffff8c723cc54000-0xffff8c723ffbffff] [ 4.064323] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.068372] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.071770] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 5.511172] Initialise system trusted keyrings [ 5.520338] Key type blacklist registered [ 5.531115] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.571115] zbud: loaded [ 5.576595] *** VALIDATE nfs *** [ 5.577789] *** VALIDATE nfs4 *** [ 5.579949] pstore: using deflate compression [ 5.602519] Platform Keyring initialized [ 5.953153] NET: Registered protocol family 38 [ 5.956246] Key type asymmetric registered [ 5.958729] Asymmetric key parser 'x509' registered [ 5.963893] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.971052] io scheduler mq-deadline registered [ 5.973967] io scheduler kyber registered [ 5.978572] io scheduler bfq registered [ 5.981720] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.987219] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.993928] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.999272] ACPI: Power Button [PWRF] [ 6.012582] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.026243] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.054568] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.072728] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.097639] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.126241] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.154585] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.159966] Non-volatile memory driver v1.3 [ 6.162199] Linux agpgart interface v0.103 [ 6.228977] virtio_blk virtio1: [vda] 136000 512-byte logical blocks (69.6 MB/66.4 MiB) [ 6.235140] vda: detected capacity change from 0 to 69632000 [ 6.280104] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.284093] vdb: detected capacity change from 0 to 1073741824 [ 6.319742] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.326701] vdc: detected capacity change from 0 to 2621440000 [ 6.391450] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.396228] vdd: detected capacity change from 0 to 2621440000 [ 6.416798] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.421023] vde: detected capacity change from 0 to 4294967296 [ 6.461541] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.467061] vdf: detected capacity change from 0 to 4294967296 [ 6.485364] libphy: Fixed MDIO Bus: probed [ 6.499918] usbcore: registered new interface driver usbserial_generic [ 6.505145] usbserial: USB Serial support registered for generic [ 6.508785] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.516210] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.524671] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.530434] mousedev: PS/2 mouse device common for all mice [ 6.540181] rtc_cmos 00:05: RTC can wake from S4 [ 6.548493] rtc_cmos 00:05: registered as rtc0 [ 6.550984] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.554356] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.558958] intel_pstate: CPU model not supported [ 6.573761] hid: raw HID events driver (C) Jiri Kosina [ 6.577902] usbcore: registered new interface driver usbhid [ 6.580182] usbhid: USB HID core driver [ 6.581771] drop_monitor: Initializing network drop monitor service [ 6.585813] Initializing XFRM netlink socket [ 6.590894] NET: Registered protocol family 10 [ 6.597616] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.604465] Segment Routing with IPv6 [ 6.607407] NET: Registered protocol family 17 [ 6.611931] mpls_gso: MPLS GSO support [ 6.620053] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.622668] RAS: Correctable Errors collector initialized. [ 6.634457] AVX version of gcm_enc/dec engaged. [ 6.641659] AES CTR mode by8 optimization enabled [ 6.802874] sched_clock: Marking stable (6802795127, 0)->(8154217739, -1351422612) [ 6.808877] registered taskstats version 1 [ 6.819359] Loading compiled-in X.509 certificates [ 6.824361] zswap: loaded using pool lzo/zbud [ 6.871613] Key type big_key registered [ 6.888939] Key type encrypted registered [ 6.895741] ima: No TPM chip found, activating TPM-bypass! [ 6.900341] ima: Allocated hash algorithm: sha1 [ 6.905833] ima: No architecture policies found [ 6.909197] evm: Initialising EVM extended attributes: [ 6.911458] evm: security.selinux [ 6.912478] evm: security.ima [ 6.913778] evm: security.capability [ 6.918435] evm: HMAC attrs: 0x1 [ 6.932265] rtc_cmos 00:05: setting system clock to 2026-06-03 01:29:33 UTC (1780450173) [ 6.945221] debug: unmapping init [mem 0xffffffffb1203000-0xffffffffb13fffff] [ 6.964907] debug: unmapping init [mem 0xffffffffaff82000-0xffffffffb0258fff] [ 6.985929] Write protecting the kernel read-only data: 28672k [ 7.001763] debug: unmapping init [mem 0xffffffffae603000-0xffffffffae7fffff] [ 7.013674] debug: unmapping init [mem 0xffffffffaef14000-0xffffffffaeffffff] [ 7.128388] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.147144] systemd[1]: Detected virtualization kvm. [ 7.151705] systemd[1]: Detected architecture x86-64. [ 7.158937] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.201725] systemd[1]: No hostname configured. [ 7.204805] systemd[1]: Set hostname to . [ 7.209813] random: systemd: uninitialized urandom read (16 bytes read) [ 7.214259] systemd[1]: Initializing machine ID from random generator. [ 7.335808] random: ln: uninitialized urandom read (6 bytes read) [ 7.606236] random: systemd: uninitialized urandom read (16 bytes read) [ 7.611640] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.622868] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 7.660169] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.466896] device-mapper: uevent: version 1.0.3 [ 9.479291] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.398386] virtio_net virtio0 ens2: renamed from eth0 [ 11.441751] random: fast init done [ 11.672219] scsi host0: ata_piix [ 11.718408] scsi host1: ata_piix [ 11.719855] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.722485] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.437438] random: crng init done [ 16.439273] random: 7 urandom warning(s) missed due to ratelimiting [ 16.545973] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.546449] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.035707] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.886184] SELinux: Disabled at runtime. [ 23.008435] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.023976] systemd[1]: Detected virtualization kvm. [ 23.027091] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.565856] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.571477] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.579272] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.586475] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.597909] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.619766] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.631195] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK [[ 25.230817] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 26.129966] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.149808] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.186507] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.442708] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.534132] EDAC sbridge: Ver: 1.1.2 [ 31.049206] Key type dns_resolver registered [ 31.469803] NFS: Registering the id_resolver key type [ 31.480288] Key type id_resolver registered [ 31.485834] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg429-server login: [ 81.931094] libcfs: loading out-of-tree module taints kernel. [ 81.994128] Key type ._llcrypt registered [ 81.998623] Key type .llcrypt registered [ 82.139520] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 103.680723] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 105.101102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.133880] alg: No test for adler32 (adler32-zlib) [ 106.749374] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 107.683659] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 109.527756] Key type lgssc registered [ 111.439859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.024865] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.677159] hrtimer: interrupt took 2783660 ns [ 178.104761] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 190.710568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 190.765857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.033858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 192.088686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 192.176283] Lustre: lustre-MDT0000: new disk, initializing [ 192.275386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.294772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.397321] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.936702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.021250] Lustre: 6513:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 208.058671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 208.062970] Lustre: Skipped 1 previous similar message [ 208.136324] Lustre: lustre-MDT0001: new disk, initializing [ 208.208032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.237545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 208.248670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 212.134871] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.688395] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 225.493199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.873732] Lustre: lustre-OST0000: new disk, initializing [ 225.879697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 225.894301] Lustre: 8418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.963763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.697712] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.986150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 231.999601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 232.081847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 246.039509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 246.183040] Lustre: lustre-OST0001: new disk, initializing [ 246.189441] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 246.199436] Lustre: 9473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 246.280879] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 252.016515] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.037433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 256.049546] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 256.149996] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 263.751499] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.290974] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.689443] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing check_logdir /tmp/testlogs/ [ 281.164405] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing yml_node [ 285.549089] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 287.727628] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 289.857487] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 291.567347] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 2 21:34:17 EDT 2026 [ 306.281704] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 307.717894] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 326.113821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 326.122923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.137810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.651312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.659275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.649493] Lustre: server umount lustre-MDT0000 complete [ 333.111821] LustreError: 10271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780450499 with bad export cookie 16629352186376917405 [ 333.113639] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 333.123864] LustreError: 10271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 337.887996] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.889273] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.896092] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 337.896101] Lustre: Skipped 3 previous similar messages [ 337.916984] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 337.925099] Lustre: Skipped 3 previous similar messages [ 338.912710] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.930039] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 339.461315] Lustre: server umount lustre-MDT0001 complete [ 343.072562] Lustre: server umount lustre-OST0000 complete [ 346.328166] Lustre: server umount lustre-OST0001 complete [ 351.750772] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 360.194247] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 392.543738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 392.912699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 392.992635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 393.109505] Lustre: lustre-MDT0000: new disk, initializing [ 393.315805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 393.369055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 397.820582] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 408.576160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.717867] Lustre: 14987:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 408.781918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 408.786847] Lustre: Skipped 1 previous similar message [ 408.843866] Lustre: lustre-MDT0001: new disk, initializing [ 408.971189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 409.010802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 409.025298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 413.468203] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 421.598655] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 425.142891] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 431.882443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 432.147899] Lustre: lustre-OST0000: new disk, initializing [ 432.151723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 432.161690] Lustre: 16731:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 432.274490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 433.904737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 433.912930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 434.033741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 437.804251] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 444.696549] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 449.019359] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 449.253150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 453.129072] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 453.315615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 459.746760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 459.751993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.753665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 459.753672] Lustre: Skipped 3 previous similar messages [ 459.761517] LustreError: Skipped 1 previous similar message [ 459.776248] Lustre: Skipped 1 previous similar message [ 461.170106] Lustre: server umount lustre-OST0000 complete [ 464.874726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.889201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 464.896586] Lustre: Skipped 1 previous similar message [ 464.900552] Lustre: Skipped 1 previous similar message [ 470.824157] Lustre: server umount lustre-MDT0000 complete [ 474.095969] LustreError: 16730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780450640 with bad export cookie 16629352186376918931 [ 474.097760] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.106344] LustreError: 16730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 474.462289] Lustre: server umount lustre-MDT0001 complete [ 477.715644] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 21:37:23 (1780450643) [ 483.909690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.301362] LustreError: 18847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.381807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.490810] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 489.440592] LustreError: 18848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.541333] LustreError: 18847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.522656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.648483] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 503.261850] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 506.525187] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 511.392783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 516.562991] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 521.301649] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 539.618500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.625069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.627036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.637439] Lustre: Skipped 1 previous similar message [ 539.642043] Lustre: Skipped 3 previous similar messages [ 543.175971] Lustre: server umount lustre-OST0000 complete [ 547.296364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 552.645729] Lustre: server umount lustre-MDT0000 complete [ 554.980880] LustreError: 18848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.002308] LustreError: 18848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 555.892278] LustreError: 18827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780450722 with bad export cookie 16629352186376919813 [ 555.905059] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.908817] LustreError: 18827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 556.175456] Lustre: server umount lustre-MDT0001 complete [ 568.529991] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 571.823607] Key type lgssc unregistered [ 572.200229] LNet: 22043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.216387] LNetError: 22043:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 572.243773] LNet: Removed LNI 192.168.204.129@tcp [ 573.522246] Key type .llcrypt unregistered [ 573.526667] Key type ._llcrypt unregistered [ 592.730654] Key type ._llcrypt registered [ 592.734077] Key type .llcrypt registered [ 592.850756] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 593.947343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 593.982904] alg: No test for adler32 (adler32-zlib) [ 595.040587] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 595.295194] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 596.999177] Key type lgssc registered [ 597.964574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 605.576277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 605.602697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.159171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 611.522560] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 617.441777] LustreError: 23418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.454773] LustreError: 23418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 618.419305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.691396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 622.112267] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 626.706071] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 629.906631] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 635.312513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 635.632580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 639.603802] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 643.007388] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 653.045464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 658.403249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.405204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 658.418713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 658.427823] Lustre: Skipped 1 previous similar message [ 660.098881] Lustre: server umount lustre-OST0000 complete [ 663.521039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.545335] Lustre: Skipped 1 previous similar message [ 663.550647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 663.560024] Lustre: Skipped 1 previous similar message [ 668.641678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 668.650432] Lustre: Skipped 1 previous similar message [ 669.330840] Lustre: server umount lustre-MDT0000 complete [ 671.937602] LustreError: 23399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780450838 with bad export cookie 18084831318278768 [ 671.948212] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 672.186813] Lustre: server umount lustre-MDT0001 complete [ 682.756315] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 685.588719] Key type lgssc unregistered [ 685.854107] LNet: 27041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.864898] LNetError: 27041:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.888505] LNet: Removed LNI 192.168.204.129@tcp [ 686.612177] Key type .llcrypt unregistered [ 686.615269] Key type ._llcrypt unregistered [ 708.569282] Key type ._llcrypt registered [ 708.571556] Key type .llcrypt registered [ 708.670478] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 709.594546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 709.636423] alg: No test for adler32 (adler32-zlib) [ 710.613388] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 710.850363] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 712.504706] Key type lgssc registered [ 713.531833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 719.620644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 719.645714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.232959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 724.844426] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 730.593442] LustreError: 28437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 730.611554] LustreError: 28437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 732.262866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 732.620584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 736.618920] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 741.758888] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 745.041192] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 757.373490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 757.770120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 763.411658] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 765.949326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 768.149775] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 776.170085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.172281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 776.187696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 780.389669] Lustre: server umount lustre-OST0000 complete [ 783.840105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 783.851881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 783.871780] Lustre: Skipped 1 previous similar message [ 783.883788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 783.892092] Lustre: Skipped 1 previous similar message [ 786.403206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.417980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 786.441635] Lustre: Skipped 1 previous similar message [ 789.810340] Lustre: server umount lustre-MDT0000 complete [ 791.524376] LustreError: 28433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 791.552856] LustreError: 28433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 792.952770] LustreError: 28417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780450959 with bad export cookie 6515915705213379653 [ 792.960763] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 792.963696] LustreError: 28417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 793.261121] Lustre: server umount lustre-MDT0001 complete [ 802.395622] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 804.632684] Key type lgssc unregistered [ 804.901575] LNet: 31955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.913873] LNetError: 31955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.933638] LNet: Removed LNI 192.168.204.129@tcp [ 805.723305] Key type .llcrypt unregistered [ 805.729275] Key type ._llcrypt unregistered [ 825.426783] Key type ._llcrypt registered [ 825.431386] Key type .llcrypt registered [ 825.548893] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 826.393523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 826.487328] alg: No test for adler32 (adler32-zlib) [ 827.508706] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 827.723908] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 829.391188] Key type lgssc registered [ 830.338518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 837.336647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 837.363749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.890539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 843.440665] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 849.379560] LustreError: 33332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.424378] LustreError: 33332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 851.886752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.450838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 855.846980] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 860.986216] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 864.775229] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 870.554849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 870.841487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 876.011425] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 880.862889] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 884.160755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 894.434839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 894.441667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 894.446468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 894.467226] Lustre: Skipped 1 previous similar message [ 899.572177] Lustre: server umount lustre-OST0000 complete [ 903.650233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 903.658199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 903.673639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 903.679470] Lustre: Skipped 1 previous similar message [ 904.678863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.698207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 904.705489] Lustre: Skipped 1 previous similar message [ 908.827709] Lustre: server umount lustre-MDT0000 complete [ 909.793624] LustreError: 33329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 909.825606] LustreError: 33329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 912.445749] LustreError: 33313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780451079 with bad export cookie 14373529493860440012 [ 912.448070] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 912.456867] LustreError: 33313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 912.659496] Lustre: server umount lustre-MDT0001 complete [ 922.866888] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 925.928862] Key type lgssc unregistered [ 926.204293] LNet: 36531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.215636] LNetError: 36531:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.233938] LNet: Removed LNI 192.168.204.129@tcp [ 927.177143] Key type .llcrypt unregistered [ 927.179714] Key type ._llcrypt unregistered [ 949.660477] Key type ._llcrypt registered [ 949.664781] Key type .llcrypt registered [ 949.762855] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 950.833696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 950.869115] alg: No test for adler32 (adler32-zlib) [ 951.883651] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 952.097054] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 953.743169] Key type lgssc registered [ 954.731534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 962.000389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 962.025266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.569920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.052096] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 973.799339] LustreError: 37928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.812234] LustreError: 37928:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 977.171141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.537972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 981.581195] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 987.000704] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 991.398232] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 998.519666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 998.926450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.152921] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1009.135381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1009.339797] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1017.351033] Lustre: lustre-OST0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 1018.337563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1018.351354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.366478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1019.361714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.364373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1019.374065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.539631] Lustre: server umount lustre-OST0000 complete [ 1043.938890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1043.945353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.953652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1043.957084] Lustre: Skipped 1 previous similar message [ 1046.856802] Lustre: server umount lustre-MDT0000 complete [ 1049.991942] LustreError: 37909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780451216 with bad export cookie 11440376025178491987 [ 1049.998187] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1050.001948] LustreError: 37909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1056.395061] Lustre: server umount lustre-MDT0001 complete [ 1065.228783] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1067.515623] Key type lgssc unregistered [ 1067.808609] LNet: 41279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.817803] LNetError: 41279:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.836509] LNet: Removed LNI 192.168.204.129@tcp [ 1068.575581] Key type .llcrypt unregistered [ 1068.577847] Key type ._llcrypt unregistered [ 1086.769702] Key type ._llcrypt registered [ 1086.771400] Key type .llcrypt registered [ 1086.909484] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1088.148427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1088.170412] alg: No test for adler32 (adler32-zlib) [ 1089.284681] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1089.542305] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1091.271255] Key type lgssc registered [ 1092.415031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1100.674277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1100.693609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.243224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1106.920197] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1111.526544] LustreError: 42654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1111.549649] LustreError: 42654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1115.270942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.727500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1119.736314] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1124.648298] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1127.875801] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1133.182406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1133.464318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1137.687664] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1141.405459] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1146.808342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1151.457576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1151.478343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.501291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1151.983606] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.003657] Lustre: Skipped 1 previous similar message [ 1152.018552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1152.027810] Lustre: Skipped 2 previous similar messages [ 1156.785820] Lustre: server umount lustre-MDT0000 complete [ 1157.089032] LustreError: 44773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.107400] LustreError: 44773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1159.589850] LustreError: 42635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780451326 with bad export cookie 23385937831309342 [ 1159.598413] LustreError: 42635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1159.600796] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1162.209914] LustreError: 42650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.211276] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.235897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1162.249946] Lustre: Skipped 1 previous similar message [ 1164.259621] LustreError: 42648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.267533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1164.278483] LustreError: 42648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1165.979742] Lustre: server umount lustre-MDT0001 complete [ 1186.540556] Lustre: server umount lustre-OST0000 complete [ 1201.786040] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1204.939477] Key type lgssc unregistered [ 1205.215729] LNet: 46160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.227102] LNetError: 46160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1205.253066] LNet: Removed LNI 192.168.204.129@tcp [ 1206.167134] Key type .llcrypt unregistered [ 1206.174207] Key type ._llcrypt unregistered [ 1229.802389] Key type ._llcrypt registered [ 1229.806440] Key type .llcrypt registered [ 1229.922388] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1230.829329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1230.883309] alg: No test for adler32 (adler32-zlib) [ 1231.964922] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1232.171921] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1233.864112] Key type lgssc registered [ 1234.953853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1242.498335] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1242.525787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1258.081626] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1263.200783] LustreError: 47528:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 1289.889518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.583153] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1300.436399] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1466.256842] Lustre: server umount lustre-OST0000 complete [ 1479.254797] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1481.373837] Key type lgssc unregistered [ 1481.602324] LNet: 49072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.612484] LNetError: 49072:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.626533] LNet: Removed LNI 192.168.204.129@tcp [ 1482.381352] Key type .llcrypt unregistered [ 1482.383081] Key type ._llcrypt unregistered [ 1504.439602] Key type ._llcrypt registered [ 1504.441388] Key type .llcrypt registered [ 1504.588259] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1505.619931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1505.670940] alg: No test for adler32 (adler32-zlib) [ 1506.888736] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1507.251024] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1509.063367] Key type lgssc registered [ 1511.407912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1520.322241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1520.358973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1521.976501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1526.231583] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1532.401099] LustreError: 50472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.441225] LustreError: 50472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1535.086291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.630169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1540.668955] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1547.421499] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1551.472595] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1558.020181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1558.366927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1563.830615] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1566.599274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1569.074743] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1571.813630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1571.816788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.822152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1571.832592] Lustre: Skipped 1 previous similar message [ 1573.322886] Lustre: server umount lustre-OST0000 complete [ 1576.929454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.942479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1576.949643] Lustre: Skipped 1 previous similar message [ 1581.536342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1581.545222] LustreError: Skipped 1 previous similar message [ 1581.546902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.557690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1582.842283] Lustre: server umount lustre-MDT0000 complete [ 1586.299546] LustreError: 51191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780451752 with bad export cookie 14329448807284343035 [ 1586.309640] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1586.317636] LustreError: 51191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1586.594954] Lustre: server umount lustre-MDT0001 complete [ 1596.164345] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1598.602716] Key type lgssc unregistered [ 1598.875784] LNet: 53644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.885197] LNetError: 53644:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1598.907664] LNet: Removed LNI 192.168.204.129@tcp [ 1599.695687] Key type .llcrypt unregistered [ 1599.702491] Key type ._llcrypt unregistered [ 1619.177507] Key type ._llcrypt registered [ 1619.179508] Key type .llcrypt registered [ 1619.287790] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1620.168176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1620.183273] alg: No test for adler32 (adler32-zlib) [ 1621.170627] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1621.370535] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1623.015253] Key type lgssc registered [ 1624.024701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1630.394736] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1630.413583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1646.047467] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1651.167208] LustreError: 54993:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 1677.855361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.378195] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1690.504281] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1698.023735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.676967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1698.800655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1703.682601] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1703.912771] LustreError: 55019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1703.937268] LustreError: 55019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1709.031615] LustreError: 55018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1713.122505] LustreError: 55918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1713.145447] LustreError: 55918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1713.641931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1714.115201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1718.821626] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1725.073969] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1729.797512] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1734.624322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1734.633772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.665583] LustreError: Skipped 1 previous similar message [ 1734.667625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1734.701589] Lustre: Skipped 1 previous similar message [ 1739.749621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1739.756656] Lustre: Skipped 1 previous similar message [ 1744.877099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1744.885887] Lustre: Skipped 1 previous similar message [ 1746.399187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1746.629126] Lustre: server umount lustre-OST0000 complete [ 1770.465351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.466508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1770.485254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.496521] Lustre: Skipped 2 previous similar messages [ 1774.627792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1774.635029] Lustre: Skipped 1 previous similar message [ 1784.287191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1784.635526] Lustre: server umount lustre-MDT0000 complete [ 1784.801929] LustreError: 55922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1784.830309] LustreError: 55922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1788.743485] LustreError: 55001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780451955 with bad export cookie 17210881964187260027 [ 1788.748851] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1788.767476] LustreError: 55001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1803.231249] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1803.440806] Lustre: server umount lustre-MDT0001 complete [ 1813.589063] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1816.464477] Key type lgssc unregistered [ 1816.820829] LNet: 58384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1816.830285] LNetError: 58384:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1816.872740] LNet: Removed LNI 192.168.204.129@tcp [ 1817.633690] Key type .llcrypt unregistered [ 1817.636290] Key type ._llcrypt unregistered [ 1841.061235] Key type ._llcrypt registered [ 1841.063539] Key type .llcrypt registered [ 1841.154991] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1842.181397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1842.250242] alg: No test for adler32 (adler32-zlib) [ 1843.292264] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1843.521382] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1845.175406] Key type lgssc registered [ 1846.349977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1853.020814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1853.040173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.487686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.546205] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1864.685540] LustreError: 59783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1864.742330] LustreError: 59783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1869.794815] LustreError: 59782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.872901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.934896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1878.475593] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1885.161311] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1889.507927] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1897.805411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.269675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.342167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1906.039218] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1912.240577] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1931.232447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1931.234436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.244876] LustreError: Skipped 1 previous similar message [ 1931.272502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1931.290032] Lustre: Skipped 1 previous similar message [ 1931.462094] Lustre: server umount lustre-OST0000 complete [ 1936.358376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1936.363916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1936.374766] Lustre: Skipped 2 previous similar messages [ 1936.399304] Lustre: Skipped 1 previous similar message [ 1941.474432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1941.479078] Lustre: Skipped 1 previous similar message [ 1941.744749] Lustre: server umount lustre-MDT0000 complete [ 1945.281035] LustreError: 59765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780452111 with bad export cookie 10190064927645971335 [ 1945.290766] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1945.294331] LustreError: 59765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1945.757430] Lustre: server umount lustre-MDT0001 complete [ 1955.565187] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1958.344685] Key type lgssc unregistered [ 1958.605351] LNet: 62993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.613117] LNetError: 62993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.653411] LNet: Removed LNI 192.168.204.129@tcp [ 1960.529966] Key type .llcrypt unregistered [ 1960.534600] Key type ._llcrypt unregistered [ 1989.506613] Key type ._llcrypt registered [ 1989.510495] Key type .llcrypt registered [ 1989.653546] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1990.949288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1991.007823] alg: No test for adler32 (adler32-zlib) [ 1992.264633] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1992.603213] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1994.295182] Key type lgssc registered [ 1995.924238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2006.380240] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:02:52 (1780452172) [ 2014.269504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2014.300699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.930403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.985696] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2025.449677] LustreError: 64682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2025.475646] LustreError: 64682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2028.986318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.350651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2034.571341] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2040.174816] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2044.365956] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2051.351723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2051.717232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.931536] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2060.999216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2063.809640] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2070.419617] Lustre: Failing over lustre-MDT0000 [ 2070.500163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2070.509429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.531406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2070.767470] Lustre: server umount lustre-MDT0000 complete [ 2071.525221] LustreError: 64682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.530435] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.564412] Lustre: Skipped 1 previous similar message [ 2076.644401] LustreError: 65633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.669090] LustreError: 65633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2080.132994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.270509] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2080.413849] Lustre: *** cfs_fail_loc=135, val=0*** [ 2080.521614] LustreError: 67427:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2080.528342] LustreError: 67427:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.129@tcp: cfg command failed: rc = -2 [ 2080.553809] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2080.553809] [ 2080.570958] LustreError: MGC192.168.204.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2080.610893] LustreError: 67415:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2080.622714] LustreError: 67415:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2080.630747] LustreError: 67415:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2080.637519] LustreError: 67415:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2080.692877] Lustre: server umount lustre-MDT0000 complete [ 2080.695047] LustreError: 67415:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2081.764388] LustreError: 64683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.809938] LustreError: 64683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2086.882070] LustreError: 64677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.911498] LustreError: 64677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2087.620552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.784973] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2087.805966] LustreError: 64675:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2087.975592] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (not set up) [ 2088.161245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.273887] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2092.013899] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2093.559042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2093.628164] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2093.657850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2093.903358] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2103.785103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.800355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2103.825588] Lustre: Skipped 1 previous similar message [ 2103.839678] Lustre: Skipped 1 previous similar message [ 2108.906263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2108.915207] Lustre: Skipped 1 previous similar message [ 2109.376751] Lustre: server umount lustre-OST0000 complete [ 2114.016045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2114.018056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.053587] Lustre: Skipped 1 previous similar message [ 2114.067179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.256924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.269719] Lustre: Skipped 4 previous similar messages [ 2128.352185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2128.584884] Lustre: server umount lustre-MDT0000 complete [ 2129.375950] LustreError: 65633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.386023] LustreError: 65633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2132.207335] LustreError: 64664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780452298 with bad export cookie 8981923737862344186 [ 2132.231178] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.580415] Lustre: server umount lustre-MDT0001 complete [ 2144.160444] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2146.751813] Key type lgssc unregistered [ 2147.010159] LNet: 69175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.016635] LNetError: 69175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.068658] LNet: Removed LNI 192.168.204.129@tcp [ 2148.811975] Key type .llcrypt unregistered [ 2148.817646] Key type ._llcrypt unregistered [ 2174.040679] Key type ._llcrypt registered [ 2174.043102] Key type .llcrypt registered [ 2174.161593] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2175.132660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2175.192506] alg: No test for adler32 (adler32-zlib) [ 2176.372444] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 2176.657105] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2178.407237] Key type lgssc registered [ 2179.472405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2187.539772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2187.553501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.136326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.704342] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2198.498655] LustreError: 70574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.523320] LustreError: 70574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2202.457155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.849692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2207.054497] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2212.905599] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2216.396738] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2222.388423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2222.741355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.985879] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2233.568220] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2234.047245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2239.887208] Lustre: Failing over lustre-MDT0000 [ 2240.152730] Lustre: server umount lustre-MDT0000 complete [ 2241.542631] LustreError: 70569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.064470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2244.079563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.101439] LustreError: 70573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.586500] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2246.641649] LustreError: 72189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.654899] LustreError: 72189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2249.768966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.978613] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2250.157992] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2250.275546] LustreError: 73321:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2250.289206] LustreError: 73321:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.129@tcp: cfg command failed: rc = -9 [ 2250.301196] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2250.301196] [ 2250.311068] LustreError: MGC192.168.204.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2250.323138] LustreError: 73310:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2250.340700] LustreError: 73310:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 2250.353066] LustreError: 73310:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2250.363429] LustreError: 73310:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2250.411763] Lustre: server umount lustre-MDT0000 complete [ 2250.415845] LustreError: 73310:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2251.760294] LustreError: 70570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.787969] LustreError: 70570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2257.240258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2257.398733] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2257.416083] LustreError: 70565:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2257.703812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2257.763821] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2258.007175] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2262.082653] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2263.037202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2263.080052] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2263.136755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2273.261824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.276564] Lustre: Skipped 1 previous similar message [ 2273.282093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2277.338984] Lustre: server umount lustre-OST0000 complete [ 2283.488901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2283.495479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2283.514195] Lustre: Skipped 1 previous similar message [ 2283.539461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2283.539472] Lustre: Skipped 1 previous similar message [ 2287.866559] Lustre: server umount lustre-MDT0000 complete [ 2288.609134] LustreError: 71588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.635838] LustreError: 71588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2292.055696] LustreError: 74335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780452458 with bad export cookie 8569468703245861520 [ 2292.060724] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2292.073967] LustreError: 74335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2292.481687] Lustre: server umount lustre-MDT0001 complete [ 2304.385172] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2307.954696] Key type lgssc unregistered [ 2308.279854] LNet: 75071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.294734] LNetError: 75071:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.356082] LNet: Removed LNI 192.168.204.129@tcp [ 2310.361433] Key type .llcrypt unregistered [ 2310.364312] Key type ._llcrypt unregistered [ 2324.671445] Key type ._llcrypt registered [ 2324.674551] Key type .llcrypt registered [ 2324.805449] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2325.060490] Key type .llcrypt unregistered [ 2325.064575] Key type ._llcrypt unregistered [ 2355.346583] Key type ._llcrypt registered [ 2355.356185] Key type .llcrypt registered [ 2355.596683] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2357.083770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2357.315727] alg: No test for adler32 (adler32-zlib) [ 2358.535899] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 2358.952849] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2360.663159] Key type lgssc registered [ 2361.547751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2369.504311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2369.519936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.017025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.473902] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2381.284238] LustreError: 76940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.302746] LustreError: 76940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2384.885505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.412122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2390.132147] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2396.316982] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2400.497805] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2407.309309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2407.849694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.857073] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2416.143443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2421.794735] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2431.473064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.477198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2431.504232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2431.518626] Lustre: Skipped 1 previous similar message [ 2436.576752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2436.586104] Lustre: Skipped 1 previous similar message [ 2437.066111] Lustre: server umount lustre-OST0000 complete [ 2441.704919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.706638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2441.708259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2441.727145] Lustre: Skipped 2 previous similar messages [ 2446.827404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2446.836234] Lustre: Skipped 2 previous similar messages [ 2447.272871] Lustre: server umount lustre-MDT0000 complete [ 2450.712162] LustreError: 76920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780452617 with bad export cookie 6244665026918890202 [ 2450.720038] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2450.724757] LustreError: 76920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2451.050159] Lustre: server umount lustre-MDT0001 complete [ 2461.132156] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2463.726781] Key type lgssc unregistered [ 2464.003495] LNet: 80115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.012955] LNetError: 80115:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2464.060514] LNet: Removed LNI 192.168.204.129@tcp [ 2464.799560] Key type .llcrypt unregistered [ 2464.802565] Key type ._llcrypt unregistered [ 2490.327439] Key type ._llcrypt registered [ 2490.329326] Key type .llcrypt registered [ 2490.545466] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2491.310733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2491.378098] alg: No test for adler32 (adler32-zlib) [ 2492.671439] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 2493.077765] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2494.841040] Key type lgssc registered [ 2496.994760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2509.546572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2509.587346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2511.552642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.419162] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2522.090874] LustreError: 81514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.138601] LustreError: 81514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2523.124361] LustreError: 81515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.229138] LustreError: 81514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.708499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.429118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2534.488582] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2542.883948] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2546.480956] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2553.221031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2553.626371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.748190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2559.826073] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2564.927437] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2573.218705] Lustre: server umount lustre-OST0000 complete [ 2578.403657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.410353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2578.427305] Lustre: Skipped 1 previous similar message [ 2578.445637] Lustre: Skipped 1 previous similar message [ 2583.524086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2583.536737] Lustre: Skipped 1 previous similar message [ 2585.648968] Lustre: server umount lustre-MDT0000 complete [ 2588.639147] Lustre: 80824:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780452739/real 1780452739] req@00000000d264c509 x1866939927390848/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780452755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2588.643834] LustreError: 81515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.668574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.696875] LustreError: 81515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2589.158849] LustreError: 81494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780452755 with bad export cookie 8032486384097590615 [ 2589.168130] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2589.552765] Lustre: server umount lustre-MDT0001 complete [ 2600.401512] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2603.381349] Key type lgssc unregistered [ 2603.679737] LNet: 84688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2603.686908] LNetError: 84688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2603.698969] LNet: Removed LNI 192.168.204.129@tcp [ 2604.610905] Key type .llcrypt unregistered [ 2604.617285] Key type ._llcrypt unregistered [ 2635.882628] Key type ._llcrypt registered [ 2635.891118] Key type .llcrypt registered [ 2636.176981] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2637.995983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2638.043527] alg: No test for adler32 (adler32-zlib) [ 2639.398839] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 2639.613955] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2641.335410] Key type lgssc registered [ 2642.407297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2652.291765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2652.319337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.918299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2659.458269] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2664.423627] LustreError: 86086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2664.457717] LustreError: 86086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2668.657585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.291880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2674.316328] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2680.327310] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2684.789834] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2691.381892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2691.885693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.347649] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2699.129201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2704.069567] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2719.725804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.729853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2719.752364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2719.773802] Lustre: Skipped 1 previous similar message [ 2721.257272] Lustre: server umount lustre-OST0000 complete [ 2724.838391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2724.846911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2724.853222] Lustre: Skipped 2 previous similar messages [ 2724.869722] Lustre: Skipped 1 previous similar message [ 2729.958310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.077763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.084031] Lustre: Skipped 1 previous similar message [ 2739.167207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2739.391309] Lustre: server umount lustre-MDT0000 complete [ 2740.196139] LustreError: 86796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.253579] LustreError: 86067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780452909 with bad export cookie 4157894782631048851 [ 2743.266458] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2743.273973] LustreError: 86067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2743.719504] Lustre: server umount lustre-MDT0001 complete [ 2753.513646] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2756.200581] Key type lgssc unregistered [ 2756.597346] LNet: 89264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2756.624187] LNetError: 89264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2756.648471] LNet: Removed LNI 192.168.204.129@tcp [ 2757.530598] Key type .llcrypt unregistered [ 2757.533582] Key type ._llcrypt unregistered [ 2782.266796] Key type ._llcrypt registered [ 2782.269958] Key type .llcrypt registered [ 2782.425249] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2783.745162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2783.754502] alg: No test for adler32 (adler32-zlib) [ 2784.964453] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 2785.222558] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2786.919230] Key type lgssc registered [ 2788.122582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2796.556553] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2796.581043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2812.256608] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2817.375323] LustreError: 90643:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 2844.063187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.612615] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2857.483859] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2863.500932] Lustre: server umount lustre-OST0000 complete [ 2872.328702] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 22:17:17 (1780453037) [ 2879.450898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.177354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.266067] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2885.602918] LustreError: 92168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.612815] LustreError: 92168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2889.700536] LustreError: 92169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.153605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.702583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2899.719924] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2905.955474] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2910.205023] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2918.951664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2919.279732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2925.424761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2925.851761] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2932.203344] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2946.020721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2946.021343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.040317] LustreError: Skipped 1 previous similar message [ 2946.043146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2946.083967] Lustre: Skipped 1 previous similar message [ 2949.640973] Lustre: server umount lustre-OST0000 complete [ 2956.257233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.260462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2956.278332] Lustre: Skipped 1 previous similar message [ 2956.285468] Lustre: Skipped 1 previous similar message [ 2960.172973] Lustre: server umount lustre-MDT0000 complete [ 2961.377392] LustreError: 92168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.388449] LustreError: 92168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2963.836237] LustreError: 94779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780453130 with bad export cookie 9524844359906846342 [ 2963.844772] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2963.852181] LustreError: 94779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2964.269915] Lustre: server umount lustre-MDT0001 complete [ 2976.200491] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2979.263323] Key type lgssc unregistered [ 2979.589572] LNet: 95514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2979.602138] LNetError: 95514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2979.627770] LNet: Removed LNI 192.168.204.129@tcp [ 2980.616435] Key type .llcrypt unregistered [ 2980.618820] Key type ._llcrypt unregistered [ 3024.244145] Key type ._llcrypt registered [ 3024.250925] Key type .llcrypt registered [ 3024.403381] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3025.891321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3025.989105] alg: No test for adler32 (adler32-zlib) [ 3027.030510] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 3027.354387] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3029.087374] Key type lgssc registered [ 3030.279283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3038.149306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3038.164051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.608641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.008952] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3049.961831] LustreError: 97683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3049.986679] LustreError: 97683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3052.569856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.073493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3057.434886] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3064.662932] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3070.686815] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3079.752658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3080.078090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.253860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3086.267361] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3091.763380] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3104.757615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3104.769650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3104.782617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3107.808852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3107.816491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3107.820533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.847607] Lustre: Skipped 1 previous similar message [ 3109.522306] Lustre: server umount lustre-OST0000 complete [ 3114.467047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3114.475822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3114.507870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3118.057587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3118.059535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.003335] Lustre: server umount lustre-MDT0000 complete [ 3123.170577] LustreError: 97679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3123.197565] LustreError: 97679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3124.506297] LustreError: 97663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780453291 with bad export cookie 17012473831421536028 [ 3124.515076] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3124.520545] LustreError: 97663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3125.425873] Lustre: server umount lustre-MDT0001 complete [ 3137.094601] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 3140.196782] Key type lgssc unregistered [ 3140.540741] LNet: 100863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3140.552407] LNetError: 100863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3140.568066] LNet: Removed LNI 192.168.204.129@tcp [ 3141.582435] Key type .llcrypt unregistered [ 3141.585170] Key type ._llcrypt unregistered [ 3159.904798] Key type ._llcrypt registered [ 3159.909179] Key type .llcrypt registered [ 3160.067320] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3161.224776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3161.255503] alg: No test for adler32 (adler32-zlib) [ 3162.483183] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 3162.807641] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3164.479181] Key type lgssc registered [ 3165.737657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3173.849411] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3173.880156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3189.407659] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3194.527257] LustreError: 102108:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 3221.215423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3227.898094] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3233.897924] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3240.859742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3241.081315] LustreError: MGC192.168.204.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3241.091832] LustreError: 103019:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3241.105646] LustreError: 103019:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3241.114191] LustreError: 103019:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3241.126886] LustreError: 102114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780453407 with bad export cookie 14596759489034353112 [ 3241.144184] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.195463] Lustre: server umount lustre-MDT0000 complete [ 3241.197497] LustreError: 103019:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3267.235927] Lustre: server umount lustre-OST0000 complete [ 3276.842260] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 3286.210527] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3321.247599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.492356] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3321.528747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3321.598103] Lustre: lustre-MDT0000: new disk, initializing [ 3321.673537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.685802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3326.087079] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3337.040675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.215244] Lustre: 105774:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3337.280770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3337.286931] Lustre: Skipped 1 previous similar message [ 3337.369890] Lustre: lustre-MDT0001: new disk, initializing [ 3337.491350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3337.560199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3337.576716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3342.362211] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3351.967214] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3358.496870] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3368.872239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3369.651684] Lustre: lustre-OST0000: new disk, initializing [ 3369.665962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3369.685294] Lustre: 107518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3369.832285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.446733] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3379.285784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3379.301885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3379.478157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3387.792405] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3392.766557] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3392.998929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3397.811849] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3398.022313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3400.161435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.161818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3400.167160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3400.180581] Lustre: Skipped 1 previous similar message [ 3405.290370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3405.305423] Lustre: Skipped 2 previous similar messages [ 3406.416944] Lustre: server umount lustre-OST0000 complete [ 3414.499355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3414.509805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3414.547256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3415.521320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.931378] Lustre: server umount lustre-MDT0000 complete [ 3421.444727] LustreError: 105768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780453588 with bad export cookie 14596759489034353217 [ 3421.458623] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3421.465378] LustreError: 105768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3421.916087] Lustre: server umount lustre-MDT0001 complete [ 3431.602396] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 22:26:36 (1780453596) [ 3435.582662] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3453.249361] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 3463.134207] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3499.418880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.634950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3499.684611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3499.746618] Lustre: lustre-MDT0000: new disk, initializing [ 3499.814708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3499.831156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3504.368417] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3515.699346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3515.847205] Lustre: 112241:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3515.907439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3515.910230] Lustre: Skipped 1 previous similar message [ 3516.020991] Lustre: lustre-MDT0001: new disk, initializing [ 3516.138040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3516.172380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3516.183916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3520.035551] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3527.893498] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3531.735785] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3539.475637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3539.865480] Lustre: lustre-OST0000: new disk, initializing [ 3539.873065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3539.889562] Lustre: 113980:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3540.047591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.286316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3547.314551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3547.539080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3547.912822] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3556.556489] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3562.198088] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3562.430515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3567.050210] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3567.339631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3573.216133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3573.218103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.229682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3573.229695] Lustre: Skipped 2 previous similar messages [ 3573.275066] Lustre: Skipped 1 previous similar message [ 3575.788752] Lustre: server umount lustre-OST0000 complete [ 3582.945512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3582.962522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.978689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3582.992833] Lustre: Skipped 1 previous similar message [ 3586.019954] Lustre: server umount lustre-MDT0000 complete [ 3588.583079] LustreError: 112252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.606557] LustreError: 112252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3589.570160] LustreError: 113019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780453756 with bad export cookie 14596759489034354099 [ 3589.580876] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3589.583944] LustreError: 113019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3589.962612] Lustre: server umount lustre-MDT0001 complete [ 3599.643896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.258132] LustreError: 116050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3600.345889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.013981] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3605.477188] LustreError: 116051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3609.576300] LustreError: 116050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3614.299133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.189534] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3625.910987] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3630.749700] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3637.412478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3643.511498] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3649.035785] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3655.484031] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3660.258332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3660.261382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.270353] LustreError: Skipped 1 previous similar message [ 3660.272302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3660.272310] Lustre: Skipped 2 previous similar messages [ 3660.302443] Lustre: Skipped 2 previous similar messages [ 3662.222796] Lustre: server umount lustre-OST0000 complete [ 3672.153487] Lustre: server umount lustre-MDT0000 complete [ 3675.618001] LustreError: 116051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3675.640796] LustreError: 116051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3676.101782] LustreError: 116030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780453842 with bad export cookie 14596759489034354981 [ 3676.103710] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3676.110963] LustreError: 116030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3676.575874] Lustre: server umount lustre-MDT0001 complete [ 3686.634320] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 3689.567926] Key type lgssc unregistered [ 3689.885510] LNet: 119320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3689.892265] LNetError: 119320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3689.914629] LNet: Removed LNI 192.168.204.129@tcp [ 3690.702452] Key type .llcrypt unregistered [ 3690.704773] Key type ._llcrypt unregistered [ 3709.363294] Key type ._llcrypt registered [ 3709.364866] Key type .llcrypt registered [ 3709.454604] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 3726.843971] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3727.477963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3727.523294] alg: No test for adler32 (adler32-zlib) [ 3728.855568] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 3729.215673] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3730.967141] Key type lgssc registered [ 3732.208108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3767.187955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3767.204878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3768.471913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3768.505515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3768.587300] Lustre: lustre-MDT0000: new disk, initializing [ 3768.736835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3768.767772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3773.346739] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3783.390233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.624823] Lustre: 122438:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3783.723085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3783.729372] Lustre: Skipped 1 previous similar message [ 3783.903965] Lustre: lustre-MDT0001: new disk, initializing [ 3783.963630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3783.987543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3783.991605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3787.886340] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3797.188481] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3801.803261] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3809.612309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3809.872298] Lustre: lustre-OST0000: new disk, initializing [ 3809.876955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3809.890266] Lustre: 124186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3809.992293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3815.891864] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3817.525402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3817.541249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3817.700338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3824.338865] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3829.570942] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3829.727216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3834.743989] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3835.053796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3838.432539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3838.445673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3838.458819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3838.468319] Lustre: Skipped 1 previous similar message [ 3838.491600] Lustre: Skipped 1 previous similar message [ 3843.703586] Lustre: server umount lustre-OST0000 complete [ 3848.683342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3848.684685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3848.693522] Lustre: Skipped 1 previous similar message [ 3853.794171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3853.809352] Lustre: Skipped 2 previous similar messages [ 3854.268735] Lustre: server umount lustre-MDT0000 complete [ 3858.566940] LustreError: 122432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780454025 with bad export cookie 6549475947371427745 [ 3858.583596] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3858.586587] LustreError: 122432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3859.119294] Lustre: server umount lustre-MDT0001 complete [ 3868.850143] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 22:33:54 (1780454034) [ 3876.185861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3876.821209] LustreError: 126450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3876.839657] LustreError: 126450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3876.912424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3881.360630] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3881.961640] LustreError: 126451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3887.078277] LustreError: 126450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3889.694880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.134187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3894.729467] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3900.664136] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3904.233621] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3907.563992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.566082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3907.587268] Lustre: Skipped 1 previous similar message [ 3907.599845] Lustre: Skipped 1 previous similar message [ 3912.618648] Lustre: server umount lustre-MDT0000 complete [ 3912.677236] LustreError: 127015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3912.692954] LustreError: 127015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3915.896947] LustreError: 126431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780454082 with bad export cookie 6549475947371428627 [ 3915.902988] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3915.907097] LustreError: 126431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3916.171568] Lustre: server umount lustre-MDT0001 complete [ 3924.422433] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 22:34:50 (1780454090) [ 3930.360610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3945.887887] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3951.009135] LustreError: 128640:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 3976.738075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3982.357508] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3987.690181] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3989.840039] Lustre: server umount lustre-OST0000 complete [ 3997.655791] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 22:36:03 (1780454163) [ 4004.023976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4004.305502] LustreError: 129971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.318874] LustreError: 129971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4004.356809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4008.721077] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4013.540119] LustreError: 129972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4013.551082] LustreError: 129972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4016.246458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4020.461333] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4025.855423] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4029.808803] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4036.811746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4037.186790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4037.192330] Lustre: Skipped 1 previous similar message [ 4043.163070] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4049.346943] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4057.572527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4057.582905] LustreError: Skipped 1 previous similar message [ 4057.586580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4057.598691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4062.994365] Lustre: server umount lustre-MDT0000 complete [ 4064.746920] LustreError: 129967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.768847] LustreError: 129967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4067.486857] LustreError: 129953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780454234 with bad export cookie 6549475947371429292 [ 4067.503265] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4067.507938] LustreError: 129953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4067.926348] Lustre: server umount lustre-MDT0001 complete [ 4082.190690] Lustre: server umount lustre-OST0000 complete [ 4090.671678] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 22:37:36 (1780454256) [ 4098.217516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4098.836181] LustreError: 133285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4103.904657] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4113.471386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.781270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4113.785919] Lustre: Skipped 1 previous similar message [ 4118.321099] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4124.759591] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4128.678527] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4136.163259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4143.310902] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4143.736782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4148.720876] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4153.744848] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4154.056857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4158.355418] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4158.530704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4164.584952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4164.587347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4164.597610] LustreError: Skipped 1 previous similar message [ 4164.603908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4164.623834] Lustre: Skipped 3 previous similar messages [ 4164.644753] Lustre: Skipped 4 previous similar messages [ 4175.327655] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4175.446396] Lustre: server umount lustre-OST0000 complete [ 4179.938616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.065831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4185.081808] Lustre: Skipped 7 previous similar messages [ 4185.563437] Lustre: server umount lustre-MDT0000 complete [ 4188.968372] LustreError: 133265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780454355 with bad export cookie 6549475947371430363 [ 4188.972476] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4188.979671] LustreError: 133265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4189.338297] Lustre: server umount lustre-MDT0001 complete [ 4197.827584] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 22:39:23 (1780454363) [ 4203.769398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4219.361190] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4224.479746] LustreError: 137078:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 4250.207459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4250.213447] Lustre: Skipped 1 previous similar message [ 4255.991323] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4261.661604] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4268.593534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.115728] LustreError: 137103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.149384] LustreError: 137103:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4269.320271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4274.266741] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4283.232905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.801326] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4293.365398] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4297.123150] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4301.032645] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4301.235468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4305.609443] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4305.877776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4309.473295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4309.486552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4309.504143] Lustre: Skipped 1 previous similar message [ 4309.510843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4322.271121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4322.524569] Lustre: server umount lustre-MDT0000 complete [ 4325.877751] LustreError: 137084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780454492 with bad export cookie 6549475947371431385 [ 4325.882318] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4325.896021] LustreError: 137084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4326.160198] Lustre: server umount lustre-MDT0001 complete [ 4340.263160] Lustre: server umount lustre-OST0000 complete [ 4351.347857] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 22:41:57 (1780454517) [ 4358.441687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4374.049260] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4379.168420] LustreError: 140900:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 4410.419586] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4415.332911] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4421.996651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4422.472674] LustreError: 140925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4422.488881] LustreError: 140925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4422.683965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4428.068976] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4435.806485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4439.546350] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4443.848630] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4446.742534] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4453.052417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4453.253112] Lustre: lustre-OST0001: new disk, initializing [ 4453.256675] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4453.261559] Lustre: 143433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4458.886886] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4459.109104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4459.121938] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4459.288713] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4466.091281] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4470.392349] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4470.702946] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4474.858965] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4475.034668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4477.413817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4477.423411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.431269] Lustre: Skipped 2 previous similar messages [ 4477.435881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4477.440616] Lustre: Skipped 9 previous similar messages [ 4491.231162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4491.310276] Lustre: server umount lustre-OST0000 complete [ 4495.341970] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4495.349256] LustreError: Skipped 2 previous similar messages [ 4500.802869] Lustre: server umount lustre-OST0001 complete [ 4510.424457] Lustre: server umount lustre-MDT0000 complete [ 4514.168308] LustreError: 140908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780454680 with bad export cookie 6549475947371432337 [ 4514.173048] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4514.176949] LustreError: 140908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4514.482172] Lustre: server umount lustre-MDT0001 complete [ 4535.768644] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 22:45:01 (1780454701) [ 4537.168705] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4538.512755] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4539.914803] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 22:45:05 (1780454705) [ 4545.726368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.967186] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4545.991692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4546.197216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4546.203105] Lustre: Skipped 6 previous similar messages [ 4549.516541] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4556.312224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4556.403176] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4556.427556] Lustre: 146846:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4560.539036] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4566.130077] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4570.287845] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4577.456685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4577.715030] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4585.181284] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4585.735126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4591.153965] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4595.944379] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4596.215573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4600.521893] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4600.781286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4606.432564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4606.451845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4606.477411] Lustre: Skipped 5 previous similar messages [ 4606.482134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4606.489290] Lustre: Skipped 13 previous similar messages [ 4617.183377] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4617.409498] Lustre: server umount lustre-OST0000 complete [ 4640.415462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4647.020374] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4647.041155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4647.057705] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4647.067780] Lustre: Skipped 1 previous similar message [ 4648.142489] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4653.198463] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4658.096343] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4658.299549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4662.410912] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4662.636446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4666.922464] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9bbf06887800.ost_server_uuid 50 [ 4668.830583] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9bbf06887800.ost_server_uuid in FULL state after 0 sec [ 4669.050418] LustreError: 146854:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.29@tcp) returned error from blocking AST (req@00000000bfd81ec7 x1866941223657856 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d0200425/0x5ae469ac6c0a12eb lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.29@tcp remote: 0xf1e48befae2df903 expref: 5 pid: 146854 timeout: 4769 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4669.074432] LustreError: lustre-MDT0000: A client on nid 192.168.204.29@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4669.081163] LustreError: 146842:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.29@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d0200425/0x5ae469ac6c0a12eb lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.29@tcp remote: 0xf1e48befae2df903 expref: 6 pid: 146854 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4675.052773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4675.065018] LustreError: Skipped 1 previous similar message [ 4679.818874] Lustre: server umount lustre-OST0000 complete [ 4689.642533] Lustre: server umount lustre-MDT0000 complete [ 4693.474319] LustreError: 147979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4693.513915] LustreError: 147979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4693.569643] LustreError: 146838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780454860 with bad export cookie 6549475947371433520 [ 4693.576269] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4693.582648] LustreError: 146838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4699.834874] Lustre: server umount lustre-MDT0001 complete [ 4710.210650] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4712.956432] Key type lgssc unregistered [ 4713.237478] LNet: 152326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4713.249720] LNetError: 152326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4713.264683] LNet: Removed LNI 192.168.204.129@tcp [ 4714.267657] Key type .llcrypt unregistered [ 4714.274439] Key type ._llcrypt unregistered [ 4738.826382] Key type ._llcrypt registered [ 4738.832759] Key type .llcrypt registered [ 4738.923371] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4740.574484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4740.692464] alg: No test for adler32 (adler32-zlib) [ 4741.909296] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 4742.355394] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4744.223469] Key type lgssc registered [ 4745.681777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4756.245861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4756.280935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4757.983873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4762.430131] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4768.228208] LustreError: 153725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4768.258578] LustreError: 153725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4772.588973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.264019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4777.615464] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4782.708242] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4786.074692] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4791.922190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4792.258529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4797.913708] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4802.492769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4803.539025] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4809.984202] Lustre: Failing over lustre-MDT0000 [ 4810.304674] Lustre: server umount lustre-MDT0000 complete [ 4811.279288] LustreError: 153720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4812.773422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.783661] LustreError: 153720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4812.794358] Lustre: Skipped 2 previous similar messages [ 4812.827469] LustreError: 153720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4817.890432] LustreError: 153724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4817.911929] LustreError: 153724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4822.331695] LustreError: 153706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.29@tcp arrived at 1780454988 with bad export cookie 12423078312529502671 [ 4822.358577] LustreError: 153706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4823.009604] LustreError: 154891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.046620] LustreError: 154891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4829.151812] Lustre: 153035:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780454979/real 1780454979] req@000000007a902f63 x1866942285635328/t0(0) o400->MGC192.168.204.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780454995 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4829.174608] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4829.511987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.272149] LustreError: 153724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4834.287693] LustreError: 153724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4839.634369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.695700] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4844.338294] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4845.032254] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4845.045773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4846.575476] Lustre: lustre-MDT0000: Denying connection for new client f93c8515-eb1a-45ce-88ce-f59f97c09b61 (at 192.168.204.29@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4854.755571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4854.764979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4854.778866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4860.401159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4860.406958] Lustre: Skipped 3 previous similar messages [ 4860.692268] Lustre: server umount lustre-OST0000 complete [ 4863.492680] LustreError: 157102:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4863.496892] Lustre: 156467:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4863.501609] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4863.512122] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4863.518296] LustreError: 156467:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000062ff3dba x1866942285654912/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4863.529846] LustreError: 156467:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4863.535099] LustreError: 156467:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4863.540812] Lustre: lustre-MDT0000: Recovery over after 0:18, of 2 clients 0 recovered and 2 were evicted. [ 4863.833635] Lustre: server umount lustre-MDT0000 complete [ 4865.504742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4865.516491] Lustre: Skipped 1 previous similar message [ 4865.525413] LustreError: 153720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.542858] LustreError: 153720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4867.002606] LustreError: 155341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455033 with bad export cookie 12423078312529502769 [ 4867.015454] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4867.016176] LustreError: 155341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4869.382941] Lustre: server umount lustre-MDT0001 complete [ 4878.959894] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4881.302626] Key type lgssc unregistered [ 4881.570398] LNet: 157837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4881.579757] LNetError: 157837:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4881.602680] LNet: Removed LNI 192.168.204.129@tcp [ 4882.427183] Key type .llcrypt unregistered [ 4882.428973] Key type ._llcrypt unregistered [ 4903.296503] Key type ._llcrypt registered [ 4903.300927] Key type .llcrypt registered [ 4903.403647] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4904.381419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4904.489935] alg: No test for adler32 (adler32-zlib) [ 4905.583358] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 4905.809246] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4907.447314] Key type lgssc registered [ 4908.566436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4915.703919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4915.752715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4917.404781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4921.709550] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4926.947078] LustreError: 159213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.966954] LustreError: 159213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4928.954858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4929.284991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4932.644296] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4937.519561] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4941.368172] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4947.943861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4948.289381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4954.181539] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4959.373610] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4961.593521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4969.952730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4969.968255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.994445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4972.002627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4972.010017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4972.021605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4976.038197] Lustre: server umount lustre-OST0000 complete [ 4980.704259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4980.716071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4980.729936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4980.739763] Lustre: Skipped 1 previous similar message [ 4985.425686] Lustre: server umount lustre-MDT0000 complete [ 4987.360676] LustreError: 159212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4987.377049] LustreError: 159212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4989.459212] LustreError: 159191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455156 with bad export cookie 10104107254250620622 [ 4989.466367] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4989.473803] LustreError: 159191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4989.885551] Lustre: server umount lustre-MDT0001 complete [ 5000.250178] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5003.293305] Key type lgssc unregistered [ 5003.651045] LNet: 162384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5003.665855] LNetError: 162384:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5003.682067] LNet: Removed LNI 192.168.204.129@tcp [ 5004.655701] Key type .llcrypt unregistered [ 5004.658132] Key type ._llcrypt unregistered [ 5037.233982] Key type ._llcrypt registered [ 5037.238904] Key type .llcrypt registered [ 5037.413936] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5039.394276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5039.409812] alg: No test for adler32 (adler32-zlib) [ 5040.432193] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5040.814616] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5042.703184] Key type lgssc registered [ 5044.748832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5052.430158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5052.443864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5053.760994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5058.504261] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5064.177877] LustreError: 164069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5064.212059] LustreError: 164069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5068.182411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5068.494917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5072.392439] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5077.872802] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5081.205551] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5087.266514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5087.493164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5091.974283] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5096.964539] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5098.810639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5108.193331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5108.205679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5108.215900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5109.216814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5109.222937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5109.231701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.339645] Lustre: server umount lustre-OST0000 complete [ 5119.457274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5119.478378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5119.484428] Lustre: Skipped 2 previous similar messages [ 5123.001383] Lustre: server umount lustre-MDT0000 complete [ 5124.579209] LustreError: 164070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5124.602460] LustreError: 164070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5126.327400] LustreError: 164051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455292 with bad export cookie 15204193025960339633 [ 5126.336384] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5126.340617] LustreError: 164051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5126.719769] Lustre: server umount lustre-MDT0001 complete [ 5135.841786] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5139.048825] Key type lgssc unregistered [ 5139.358945] LNet: 167246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5139.364605] LNetError: 167246:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5139.387992] LNet: Removed LNI 192.168.204.129@tcp [ 5140.209789] Key type .llcrypt unregistered [ 5140.215144] Key type ._llcrypt unregistered [ 5161.625082] Key type ._llcrypt registered [ 5161.629464] Key type .llcrypt registered [ 5161.734675] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5162.535440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5162.619038] alg: No test for adler32 (adler32-zlib) [ 5163.741595] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5163.961586] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5165.687605] Key type lgssc registered [ 5166.916613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5178.087807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5178.106810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5179.709377] Lustre: *** cfs_fail_loc=135, val=0*** [ 5179.837512] LustreError: 168750:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5179.846384] LustreError: 168750:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.129@tcp: cfg command failed: rc = -2 [ 5179.868587] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5179.868587] [ 5179.878036] LustreError: MGC192.168.204.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5179.889180] LustreError: 168711:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5179.898224] LustreError: 168711:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5179.916638] LustreError: 168711:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5179.929437] LustreError: 168711:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5180.104732] Lustre: server umount lustre-MDT0000 complete [ 5180.109597] LustreError: 168711:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5186.565313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5186.915100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5190.345321] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5197.286766] LustreError: 169121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5197.316069] LustreError: 169121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5198.874350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.317666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5203.850558] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5209.557640] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5213.998052] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5217.486572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5217.527061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5217.544472] Lustre: Skipped 1 previous similar message [ 5222.884489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5222.899209] Lustre: Skipped 1 previous similar message [ 5223.887049] Lustre: server umount lustre-MDT0000 complete [ 5223.911026] LustreError: 170463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5223.932983] LustreError: 170463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5229.025082] LustreError: 169118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5229.078918] LustreError: 169118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5229.723596] LustreError: 169103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455396 with bad export cookie 8551018011839101521 [ 5229.740380] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5230.211208] Lustre: server umount lustre-MDT0001 complete [ 5240.284304] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5243.346284] Key type lgssc unregistered [ 5243.622623] LNet: 171198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5243.628451] LNetError: 171198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5243.655319] LNet: Removed LNI 192.168.204.129@tcp [ 5244.590664] Key type .llcrypt unregistered [ 5244.592737] Key type ._llcrypt unregistered [ 5270.605679] Key type ._llcrypt registered [ 5270.608964] Key type .llcrypt registered [ 5270.777027] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5271.136573] Key type .llcrypt unregistered [ 5271.139488] Key type ._llcrypt unregistered [ 5289.713432] Key type ._llcrypt registered [ 5289.715689] Key type .llcrypt registered [ 5289.797951] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5291.069691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5291.102669] alg: No test for adler32 (adler32-zlib) [ 5292.215395] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5292.534782] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5294.208364] Key type lgssc registered [ 5295.497707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5303.376080] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5303.409943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5318.943783] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5324.064071] LustreError: 173299:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 5350.751202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5358.692206] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5365.358766] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5372.387513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5372.765755] LustreError: 172633:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000040ebabed x1866942863378688/t0(0) o250->MGC192.168.204.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5373.304175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5373.382931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5378.166437] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5378.532693] LustreError: 173325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5378.554113] LustreError: 173325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5383.652684] LustreError: 173324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.704667] LustreError: 173325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.734770] LustreError: 173325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5386.725413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.062695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5390.795350] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5395.819114] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5399.474889] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5402.204443] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5412.843321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5412.848991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5412.849666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5412.858436] Lustre: Skipped 1 previous similar message [ 5412.877648] Lustre: Skipped 1 previous similar message [ 5417.956111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5417.966891] Lustre: Skipped 1 previous similar message [ 5418.508078] Lustre: server umount lustre-OST0000 complete [ 5423.075416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5423.083813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5423.094376] Lustre: Skipped 1 previous similar message [ 5423.109828] Lustre: Skipped 1 previous similar message [ 5428.193087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5428.200766] Lustre: Skipped 1 previous similar message [ 5433.329224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5433.332734] Lustre: Skipped 1 previous similar message [ 5436.383129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5436.534916] Lustre: server umount lustre-MDT0000 complete [ 5438.436981] LustreError: 174230:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.452792] LustreError: 174230:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5441.003155] LustreError: 173306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455607 with bad export cookie 17228178281324542455 [ 5441.008087] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5441.017401] LustreError: 173306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5441.397089] Lustre: server umount lustre-MDT0001 complete [ 5452.044454] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5454.585323] Key type lgssc unregistered [ 5454.911306] LNet: 176910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5454.921950] LNetError: 176910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5454.938028] LNet: Removed LNI 192.168.204.129@tcp [ 5455.705586] Key type .llcrypt unregistered [ 5455.712069] Key type ._llcrypt unregistered [ 5481.016500] Key type ._llcrypt registered [ 5481.020222] Key type .llcrypt registered [ 5481.221656] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5482.549221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5482.636805] alg: No test for adler32 (adler32-zlib) [ 5484.073818] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5484.271265] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5486.063218] Key type lgssc registered [ 5487.591572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5496.892306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5496.915792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5498.587304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5503.139428] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5509.093397] LustreError: 178315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5509.113174] LustreError: 178315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5511.526519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5512.026122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5516.739710] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5522.859314] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5527.029350] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5534.361211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5534.931554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.729737] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5545.218637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5545.844268] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5550.156661] Lustre: Failing over lustre-MDT0000 [ 5550.365118] Lustre: server umount lustre-MDT0000 complete [ 5550.578568] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.582186] LustreError: 178312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5550.601076] Lustre: Skipped 1 previous similar message [ 5552.634142] LustreError: 178310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5552.654182] LustreError: 178310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5555.681408] LustreError: 178312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5555.691819] LustreError: 178312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5560.801865] LustreError: 178316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5560.819701] LustreError: 178316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5567.368571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5567.496930] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5567.756393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5567.815645] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5567.978208] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5571.585804] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5573.106496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5573.139611] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 5573.194064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5575.509118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5582.384729] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5592.148439] Lustre: server umount lustre-OST0000 complete [ 5598.692220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5598.700375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5598.711061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5598.741622] Lustre: Skipped 2 previous similar messages [ 5603.810436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5603.819934] Lustre: Skipped 2 previous similar messages [ 5607.170790] Lustre: server umount lustre-MDT0000 complete [ 5608.953380] LustreError: 178315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5608.991390] LustreError: 178315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5609.951187] Lustre: 177617:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780455760/real 1780455760] req@000000006519cec9 x1866943063707008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780455776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5609.952675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.992699] Lustre: 177617:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5611.964739] LustreError: 180605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455778 with bad export cookie 11225662379884538494 [ 5611.979430] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5612.416747] Lustre: server umount lustre-MDT0001 complete [ 5622.406250] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5625.261112] Key type lgssc unregistered [ 5625.679838] LNet: 182826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5625.688268] LNetError: 182826:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5625.721887] LNet: Removed LNI 192.168.204.129@tcp [ 5626.581540] Key type .llcrypt unregistered [ 5626.590071] Key type ._llcrypt unregistered [ 5650.177992] Key type ._llcrypt registered [ 5650.182569] Key type .llcrypt registered [ 5650.279550] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5650.989573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5651.067706] alg: No test for adler32 (adler32-zlib) [ 5652.117675] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5652.303836] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5653.959255] Key type lgssc registered [ 5655.026258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5662.070919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5662.101720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5663.669511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5667.712925] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5673.952958] LustreError: 184223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5673.967459] LustreError: 184223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5675.480079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.855446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5680.121215] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5685.395152] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5689.281339] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5696.247756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5696.660978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.671327] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5708.979506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5710.928961] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5714.528741] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5721.581781] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5738.121426] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5745.123207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.147910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5745.168177] Lustre: Skipped 1 previous similar message [ 5749.177304] Lustre: server umount lustre-OST0000 complete [ 5755.363411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.385805] Lustre: Skipped 1 previous similar message [ 5755.393665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5755.401994] Lustre: Skipped 1 previous similar message [ 5759.577406] Lustre: server umount lustre-MDT0000 complete [ 5760.491070] LustreError: 186330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5760.517099] LustreError: 186330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5764.274795] LustreError: 186419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455930 with bad export cookie 6223452686774895659 [ 5764.278305] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5764.283216] LustreError: 186419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5764.695800] Lustre: server umount lustre-MDT0001 complete [ 5775.761490] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5778.500026] Key type lgssc unregistered [ 5778.912917] LNet: 187731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5778.924852] LNetError: 187731:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5778.947909] LNet: Removed LNI 192.168.204.129@tcp [ 5779.752460] Key type .llcrypt unregistered [ 5779.756533] Key type ._llcrypt unregistered [ 5797.985772] Key type ._llcrypt registered [ 5797.988583] Key type .llcrypt registered [ 5798.077653] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5798.996052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5799.056245] alg: No test for adler32 (adler32-zlib) [ 5800.044323] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5800.256953] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5801.927173] Key type lgssc registered [ 5802.843982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5809.457369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5809.484711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5810.889556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5814.882751] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5821.409732] LustreError: 188978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5821.433223] LustreError: 188978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5823.061425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5823.333337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5827.160796] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5832.556171] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5836.225142] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5843.055223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5843.429382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5849.154570] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5854.449469] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5855.764188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5860.605847] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5873.345789] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5881.371990] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5890.148647] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5901.801133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5901.802810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5901.813675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5901.836025] Lustre: Skipped 1 previous similar message [ 5905.226546] Lustre: server umount lustre-OST0000 complete [ 5910.496618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5910.503422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.520925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5910.530223] Lustre: Skipped 1 previous similar message [ 5912.034241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5912.036402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5915.684594] Lustre: server umount lustre-MDT0000 complete [ 5917.155856] LustreError: 189999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5917.174486] LustreError: 189999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5919.219710] LustreError: 188958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780456085 with bad export cookie 12839763800046293509 [ 5919.222608] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5919.229694] LustreError: 188958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5919.535458] Lustre: server umount lustre-MDT0001 complete [ 5930.649852] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5933.637687] Key type lgssc unregistered [ 5933.965730] LNet: 193638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5933.975854] LNetError: 193638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5933.997920] LNet: Removed LNI 192.168.204.129@tcp [ 5934.980473] Key type .llcrypt unregistered [ 5934.982373] Key type ._llcrypt unregistered [ 5958.951186] Key type ._llcrypt registered [ 5958.954978] Key type .llcrypt registered [ 5959.102839] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5960.036250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5960.087101] alg: No test for adler32 (adler32-zlib) [ 5961.194689] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5961.532085] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5963.207182] Key type lgssc registered [ 5964.029147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5972.462129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5972.497950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5974.192232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5978.531868] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5983.715001] LustreError: 195036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5983.728220] LustreError: 195036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5986.771809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5987.207854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5991.511576] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5996.702553] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5999.987667] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6005.983318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6006.338805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6012.129510] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6017.173151] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6018.609891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6032.865733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6032.876637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6032.899228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6033.890320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.894792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6033.897941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6036.417750] Lustre: server umount lustre-OST0000 complete [ 6043.616730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6043.630948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6043.654335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6043.662427] Lustre: Skipped 1 previous similar message [ 6049.250096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6049.264824] Lustre: Skipped 3 previous similar messages [ 6054.367145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6054.368210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6054.389551] Lustre: Skipped 1 previous similar message [ 6054.616761] Lustre: server umount lustre-MDT0000 complete [ 6058.499727] LustreError: 197190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6058.530915] LustreError: 197190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6058.936750] LustreError: 195018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780456225 with bad export cookie 15900567138665605534 [ 6058.937827] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6058.946063] LustreError: 195018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6059.465305] Lustre: server umount lustre-MDT0001 complete [ 6072.791262] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6076.625823] Key type lgssc unregistered [ 6077.003575] LNet: 198365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6077.006210] LNetError: 198365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6078.054716] LNet: Removed LNI 192.168.204.129@tcp [ 6079.328387] Key type .llcrypt unregistered [ 6079.335495] Key type ._llcrypt unregistered [ 6105.570852] Key type ._llcrypt registered [ 6105.575159] Key type .llcrypt registered [ 6105.769674] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6107.549511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6107.683330] alg: No test for adler32 (adler32-zlib) [ 6108.805652] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 6109.082347] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 6110.799776] Key type lgssc registered [ 6111.955021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6118.990543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6119.010213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6120.342720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6124.274282] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6129.636497] LustreError: 199615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6129.658601] LustreError: 199615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6131.727969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6132.113254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6136.022214] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6140.686381] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6143.955300] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6150.189858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6150.447485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6155.296542] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6161.165676] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6164.797311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6183.397634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6183.410988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6183.441605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6185.444038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6185.448941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6185.449847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6187.136870] Lustre: server umount lustre-OST0000 complete [ 6193.635039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6193.654625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6193.664418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6193.668019] Lustre: Skipped 1 previous similar message [ 6195.682265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6195.695219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6198.855507] Lustre: server umount lustre-MDT0000 complete [ 6200.801724] LustreError: 201233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6200.820523] LustreError: 201233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6202.433134] LustreError: 201236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780456368 with bad export cookie 3010853268418361198 [ 6202.435917] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6202.452688] LustreError: 201236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6202.895624] Lustre: server umount lustre-MDT0001 complete [ 6212.579951] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6215.459858] Key type lgssc unregistered [ 6215.803785] LNet: 202943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6215.824374] LNetError: 202943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6215.851078] LNet: Removed LNI 192.168.204.129@tcp [ 6216.719171] Key type .llcrypt unregistered [ 6216.723535] Key type ._llcrypt unregistered [ 6234.018992] Key type ._llcrypt registered [ 6234.026722] Key type .llcrypt registered [ 6234.118562] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6235.157417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6235.199773] alg: No test for adler32 (adler32-zlib) [ 6236.159197] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 6236.405534] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 6238.072288] Key type lgssc registered [ 6239.101591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6245.923829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6245.943583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6247.479939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6250.984308] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6257.632996] LustreError: 204194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6257.665178] LustreError: 204194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6258.134717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6258.515454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6262.346616] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6267.529569] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6271.464116] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6277.158488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6277.589908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6283.320195] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6288.852851] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6292.983964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6301.344236] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6301.435673] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6301.634431] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6306.810742] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6311.929781] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6313.347200] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6325.079857] Lustre: Permanently deactivating lustre-OST0001 [ 6325.096905] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6332.529589] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6339.518771] Lustre: Permanently reactivating lustre-OST0001 [ 6339.545569] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6339.549603] Lustre: Skipped 2 previous similar messages [ 6347.819578] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6347.841325] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6347.849804] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6347.866821] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6347.894526] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6352.364228] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.369126] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6352.379970] Lustre: Skipped 2 previous similar messages [ 6357.474532] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6357.481717] Lustre: Skipped 2 previous similar messages [ 6358.162871] Lustre: server umount lustre-OST0001 complete [ 6362.597122] LustreError: 205787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6362.597970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6362.609132] LustreError: 205787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6362.624358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6367.725687] Lustre: server umount lustre-OST0000 complete [ 6371.298561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6371.306951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.318425] Lustre: Skipped 1 previous similar message [ 6371.322476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6371.327979] Lustre: Skipped 1 previous similar message [ 6377.583253] Lustre: server umount lustre-MDT0000 complete [ 6377.952340] LustreError: 204904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.980441] LustreError: 204904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6381.667481] LustreError: 204173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780456548 with bad export cookie 4635099772840978779 [ 6381.668819] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6381.687857] LustreError: 204173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6382.183701] Lustre: server umount lustre-MDT0001 complete [ 6395.594873] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6398.425547] Key type lgssc unregistered [ 6398.758981] LNet: 208792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6398.773499] LNetError: 208792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6398.803434] LNet: Removed LNI 192.168.204.129@tcp [ 6399.831809] Key type .llcrypt unregistered [ 6399.833693] Key type ._llcrypt unregistered [ 6436.894817] Key type ._llcrypt registered [ 6436.897875] Key type .llcrypt registered [ 6437.040295] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6438.136951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6438.208123] alg: No test for adler32 (adler32-zlib) [ 6439.306526] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 6439.562723] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 6441.271125] Key type lgssc registered [ 6443.374556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6454.094729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6454.127784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6455.426768] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6455.449138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6455.773608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6459.926963] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6470.260509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.333722] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6470.353752] Lustre: 210982:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6470.631846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6474.636841] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6480.247591] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6484.144599] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6490.847491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6491.167649] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6491.513802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6497.966053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6498.202157] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6504.271168] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6507.371222] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6516.443310] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6525.077504] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6533.081633] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6541.978908] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6566.348080] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6566.357372] Lustre: Skipped 2 previous similar messages [ 6598.808261] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6598.813601] Lustre: Skipped 3 previous similar messages [ 6666.631976] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6666.647874] Lustre: Skipped 8 previous similar messages [ 6687.714038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6687.724812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6687.738025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6692.321306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6692.326595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6692.336261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6693.488375] Lustre: server umount lustre-OST0000 complete [ 6697.956742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6697.993201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6698.001529] Lustre: Skipped 2 previous similar messages [ 6703.072239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6703.081164] Lustre: Skipped 1 previous similar message [ 6708.201948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6708.213209] Lustre: Skipped 1 previous similar message [ 6712.288301] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6712.515308] Lustre: server umount lustre-MDT0000 complete [ 6716.742577] LustreError: 210973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780456883 with bad export cookie 8787551183694940501 [ 6716.743504] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6716.760132] LustreError: 210973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6717.235753] Lustre: server umount lustre-MDT0001 complete [ 6728.716804] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6732.494936] Key type lgssc unregistered [ 6733.036388] LNet: 215422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6733.057639] LNetError: 215422:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6734.121261] LNet: Removed LNI 192.168.204.129@tcp [ 6735.271994] Key type .llcrypt unregistered [ 6735.274777] Key type ._llcrypt unregistered [ 6765.581309] Key type ._llcrypt registered [ 6765.587190] Key type .llcrypt registered [ 6765.750256] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6766.941308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6767.010434] alg: No test for adler32 (adler32-zlib) [ 6768.122993] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 6768.401944] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 6770.095186] Key type lgssc registered [ 6771.427462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6778.732668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6778.748942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6780.267517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6784.334793] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6789.603660] LustreError: 216822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6789.642981] LustreError: 216822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6793.843359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6794.289733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6798.101193] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6802.847680] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6805.804725] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6812.039726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6812.315980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6818.550622] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6824.627135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6824.778757] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6845.352385] Lustre: server umount lustre-OST0000 complete [ 6850.529871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6850.530292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6850.536491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6850.597855] Lustre: Skipped 1 previous similar message [ 6855.650035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6855.658948] Lustre: Skipped 2 previous similar messages [ 6858.852900] Lustre: server umount lustre-MDT0000 complete [ 6860.768226] LustreError: 217841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6860.771125] Lustre: 216135:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780457011/real 1780457011] req@00000000d8313a02 x1866944410062208/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780457027 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6860.807868] LustreError: 217841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6860.865719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.182849] LustreError: 219059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780457029 with bad export cookie 13337033049113036055 [ 6863.192607] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6863.210561] LustreError: 219059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6863.673677] Lustre: server umount lustre-MDT0001 complete [ 6875.867977] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6878.991399] Key type lgssc unregistered [ 6879.355724] LNet: 220156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6879.361663] LNetError: 220156:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6879.376212] LNet: Removed LNI 192.168.204.129@tcp [ 6880.181754] Key type .llcrypt unregistered [ 6880.184540] Key type ._llcrypt unregistered [ 6902.573896] Key type ._llcrypt registered [ 6902.576489] Key type .llcrypt registered [ 6902.752173] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6904.059997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6904.234679] alg: No test for adler32 (adler32-zlib) [ 6905.465488] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 6905.962177] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 6907.680321] Key type lgssc registered [ 6909.072963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7183.062886] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7186.757977] Key type lgssc unregistered [ 7187.160621] LNet: 222145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7187.176224] LNetError: 222145:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7187.217295] LNet: Removed LNI 192.168.204.129@tcp [ 7188.232090] Key type .llcrypt unregistered [ 7188.237223] Key type ._llcrypt unregistered [ 7212.911500] Key type ._llcrypt registered [ 7212.915720] Key type .llcrypt registered [ 7213.041942] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 7228.672383] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 7229.891445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7229.976270] alg: No test for adler32 (adler32-zlib) [ 7231.359988] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 7231.693624] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 7233.399247] Key type lgssc registered [ 7234.732957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7268.951172] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6976 sec ========== 23:30:34 (1780457434) [ 7270.755243] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:30:36 (1780457436) === [ 7274.460594] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:30:39 (1780457439) === [ 7313.432486] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7316.316349] Key type lgssc unregistered [ 7316.664582] LNet: 227329:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7316.671619] LNetError: 227329:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7316.685908] LNet: Removed LNI 192.168.204.129@tcp [ 7317.487378] Key type .llcrypt unregistered [ 7317.492377] Key type ._llcrypt unregistered